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EXECUTIVE  SUMMARY 


Cueing  Synergy 


Background 

Landing  maneuvers  in  desert  conditions  pose  a  significant  risk  due  to  the  likelihood  of  partial 
or  total  visibility  loss  caused  by  airborne  dust  or  sand  stirred  up  by  the  helicopter’s  rotor 
downwash;  this  condition  is  termed  brownout.  Recent  empirical  evidence  indicates  there  is  a 
potential  benefit  to  supplementing  visual  displays  and  symbology  with  tactile  and  aural  cueing 
displays,  thereby  enhancing  pilot  situational  awareness.  How  to  best  display  and  synergize  this 
information  is  an  open  question. 

Purpose 

This  test  sought  to  detennine  optimized  cueing  display  configurations  to  facilitate  helicopter 
operations  in  Degraded  Visual  Environments  (DVE).  The  test  evaluated  three  visual  symbology 
sets,  as  well  as  two  non-visual  cueing  technologies  (tactile  and  aural  cues)  for  their  synergy, 
compatibility,  benefit,  or  conflict  when  used  in  DVE. 

Methods 

Eight  test  pilots  evaluated  aural  and  tactile  cueing  configurations  in  combination  with  three 
visual  symbology  sets.  Four  preselected  flight  tasks  derived  from  the  Aeronautical  Design 
Standard  Performance  Specifications  Handling  Qualities  Requirements  for  Military  Rotorcraft 
(ADS-33E-PRF)  were  analyzed  for  potential  beneficial  or  interactive  effects  of  the  cueing 
technologies:  approach  to  landing,  approach  to  hover,  hover,  and  sidestep.  The  test  flights  were 
flown  in  the  United  States  Army  Aeromedical  Research  Laboratory’s  (USAARL)  NUH-60FS 
research  flight  simulator  configured  with  UH-60M  cockpit  displays  with  simulated  infrared  (IR) 
sensor  imagery.  The  dust/brownout  simulation  was  created  with  a  dedicated  image  generator 
using  purpose  built  software,  which  accurately  simulates  brownout  conditions  in  the  visual 
database  (National  Training  Center,  Fort  Irwin,  California).  The  visual  symbology  sets  selected 
were  a  control  Legacy  symbology  derived  from  the  AN/AVS7  Legacy  HUD  (Head-up  Display), 
the  Brownout  Symbology  Set  (BOSS)  with  three-dimensional  (3D)  confonnal  symbology,  and 
the  Forward-Looking  Integrated  Systems  for  Helicopter  (FISH)  symbology.  Cueing  effectiveness 
was  assessed  by  recording  objective  indices  of  flight  perfonnance,  objective  biometric  measures, 
and  subjective  ratings  and  comments  by  the  test  pilots.  The  formal  subjective  rating  tools 
included  the  Bedford  Workload  Scale,  Cooper-Harper  Handling  Qualities  Rating  Scale,  Visual 
Cue  Index  (VCI),  a  demographic  and  cueing  display  ranking  questionnaire,  and  free-flow 
comments  report. 

Conclusions 

1 .  Generally,  pilots  performed  better  using  advanced  visual  symbologies  (BOSS  and/or  FISH) 
when  combined  with  a  supplemental  form  of  cueing  (aural  and/or  tactile). 

2.  Test  pilots’  preferred  supplemental  cueing  modality  was  dependent  on  the  type  of  visual 
symbology  and/or  flight  maneuver. 

3.  Research  pilots  observed  a  greater  number  of  pilot-  induced  oscillations  during  the  hover  and 
sidestep  maneuvers  when  aural  cueing  was  paired  with  either  Legacy  or  FISH  visual 
symbologies. 
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4.  Overall,  subjective  and  flight  performance  measures  indicated  that  the  BOSS  symbology  was 
the  preferred  visual  symbology  set,  although  some  attributes  of  FISH  were  preferred  when 
performing  the  sidestep  maneuver. 

5.  Pilots  preferred  aural  cues  that  provided  situational  information  over  aural  cues  that 
demanded  corrective  action  to  satisfy  a  required  perfonnance  measure. 

6.  In  general,  test  pilots  preferred  the  TSAS  cueing  display  over  the  aural  cueing  display. 

7.  Biometric  measures  of  stress  were  associated  with  performance  trends  on  selected  flight 
maneuvers.  Future  studies  should  exploit  these  measures  as  objective  correlates  of  aviator 
performance. 
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Introduction 


Background  and  significance 

Flying  in  blowing  sand,  dust,  rain,  or  snow  is  currently  a  major  problem  for  the  U.S.  military, 
costing  over  $100  million  a  year  in  aircraft  losses  and  resulting  in  the  loss  of  over  600  lives  from 
2002  to  2012  (Whittle,  2012).  Degraded  visual  environment  (DVE)  is  the  term  used  to  describe 
the  condition  when  visibility  outside  the  aircraft  is  severely  degraded  or  nonexistent.  Of  primary 
concern  is  brownout,  which  occurs  when  the  visual  environment  is  obscured  by  recirculated  dust, 
dirt,  or  sand  due  to  rotor  downwash  as  a  helicopter  takes  off,  hovers,  or  lands.  A  similar 
phenomenon  occurs  in  snow  and  is  called  whiteout  (NATO  Task  Group  HFM-162,  2012).  In 
order  to  successfully  operate  in  DVE,  pilots  must  be  able  to  detect  and  perceive  drift,  height 
above  terrain,  descent  rate,  ground  speed,  attitude,  ground  slope,  terrain  features,  landing  point 
location,  obstacle  clearance,  and  moving  obstacles  (NATO  Task  Group  HFM-162,  2012). 

Current  instrumentation  suffers  from  two  major  limitations.  First,  available  displays  do  not 
contain  sufficient  information,  i.e.,  drift,  ground  slope,  terrain  features,  landing  point  location, 
obstacle  clearance,  and  moving  obstacle  detection.  Second,  information  bandwidth  is  insufficient 
to  communicate  the  necessary  information  in  a  timely  manner.  During  an  approach  to  landing  in 
a  normal  visual  environment,  the  pilot  will  rely  on  outside  visual  references  for  information 
regarding  ground  speed,  lateral  drift,  landing  point  location,  and  the  landing  zone  environment. 
However,  once  the  pilot  enters  DVE,  he  can  no  longer  access  those  outside  visual  cues. 

Switching  to  flight  instruments  does  not  solve  the  problem  because  they  do  not  provide  these  key 
parameters.  Thus,  the  lack  of  necessary  information  in  DVE  increases  the  pilot’s  risk  of  crashing 
due  to  unrecognized  excessive  descent  rates,  unintended  drift,  and  ground  obstacle  collisions 
(Colucci,  2007). 

Strategies  being  urgently  explored  to  provide  the  necessary  information  needed  to  operate 
during  DVE  include  virtual  reality,  infrared  imagery,  advanced  visual  symbology,  tactile  cues, 
and  aural  cues.  Logically,  an  intuitive  multisensory  approach  may  provide  the  information 
necessary  for  DVE  operations.  This  report  explored  the  efficacy  of  combining  visual,  tactile, 
and  aural  displays  to  increase  infonnation  bandwidth  while  avoiding  sensory  conflict. 

Visual  cues 

During  landing,  visual  references  from  the  outside  environment  are  a  pilot’s  primary  source  of 
information  (NATO  Task  Group  HFM-162,  2012).  Visual  cues  from  the  outside  world  during 
landing  provide  pilots  with  information  regarding  aircraft  attitude  and  the  landing  zone 
environment;  once  a  pilot  enters  DVE,  he/she  is  literally  working  in  the  blind.  Because  humans 
are  primarily  visual  creatures,  it  is  understandable  that  visual  displays  would  become  the  primary 
DVE  mitigation  instrument  (Curtis,  Jentsch,  and  Wise,  2010)  acting  as  the  pilot’s  window  to  the 
world  (Stokes  and  Wickens,  1988).  Visual  displays  can  be  analog  or  digital  (figure  1). 
Traditional  analog  displays  used  simple  symbology,  such  as  the  use  of  colors  (red,  yellow,  green) 
to  indicate  safe  operating  ranges  and  performance  limitations.  As  analog  displays  were  replaced 
with  digital  (glass)  displays,  symbology  became  more  complex  in  an  effort  to  provide  pilots  with 
more  accurate  infonnation.  Digital  displays  are  used  to  relay  flight  data  (airspeed,  altitude, 
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attitude,  vertical  speed,  and  heading),  navigational  data,  aircraft  systems  data,  weather,  and  a 
variety  of  additional  information  needed  to  complete  complex  missions  to  the  pilot.  Glass 
cockpit  displays  have  made  it  possible  to  provide  pilots  with  information  in  intuitive  fonnats 
when  it  is  needed  (Mejdal,  McCauley,  &  Beringer,  2001). 


Figure  1.  Analog  (left)  and  digital  (right)  cockpits. 

Synthetic  vision  (sensed  and/or  database  Imagery) 

Pilots  rely  on  focal  vision  for  object  recognition  relative  to  the  helicopter  location.  Ambient, 
or  peripheral,  vision  provides  information  relating  to  the  helicopter’s  position,  motion,  and 
attitude.  Loss  of  focal  vision  information  in  DVE  can  be  overcome  with  synthetic  vision 
systems.  Forward-looking  Infrared  (FLIR)  displays  and  database  imagery  are  two  synthetic 
vision  systems  available  to  pilots. 

FLIR 

FLIR  refers  to  a  forward-looking  infrared  sensor  mounted  on  the  aircraft  that  provides  pilots 
with  a  view  of  the  external  world  even  when  operating  at  night  or  in  obscured  environments. 

This  capability  provides  pilots  with  cues  that  traditional  instruments  cannot  provide,  such  as 
drift,  ground  slope,  terrain  features,  obstacle  clearance,  and  landing  point  location.  Having  an 
image  of  the  external  visual  environment  provides  pilots  with  pertinent  information  necessary  for 
safe  operations  in  DVE;  however,  FLIR  imagery  has  its  limitations.  The  quality  of  the  FLIR 
image  can  be  distorted  due  to  sensor  restrictions.  Very  fine  dust  particles  can  obscure  the 
sensors  and  prevent  them  from  detecting  objects.  FLIR  imagery  relies  on  differences  in  heat  to 
detect  objects  and  produce  images.  While  useful,  FLIR  imagery  is  not  infallible — rain-soaked 
terrain  on  a  cloudy  day  may  present  little  thermal  contrast.  Also,  depending  on  the  climate,  twice 
a  day  objects  are  likely  to  have  very  small  heat  differences  making  object  detection  difficult  and 
FLIR  imagery  less  useful;  this  phenomenon  is  known  as  thermal  crossover. 

The  FLIR  image  displays  can  also  affect  the  quality  of  the  picture.  Time  delays  caused  by 
latency  and  refresh  rate  can  provide  pilots  with  inadequate  information.  Additionally,  size- 
distance  perception  can  be  affected  by  changes  in  the  field  of  view.  As  the  field  of  view 
increases,  the  external  objects  are  minified  on  the  display  making  the  objects  appear  further 
away.  FLIR  images  can  be  displayed  on  head-up  displays  (HUDs)s  or  panel-mounted  displays 
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(PMDs).  Panel-mounted  displays  require  pilots  to  reacquire  a  scan  as  their  eyes  move  from 
inside  the  cockpit  to  the  external  environment. 

For  the  purpose  of  this  study,  it  was  detennined  that  a  PMD  with  real  time  FLIR  imagery 
would  be  paired  with  the  various  visual  symbology  sets.  Henceforth,  any  reference  to  infrared 
(IR)  or  IR  scene  refers  to  a  symbology  set  over  the  simulated  IR  sensor  imagery.  The  display 
presents  a  sensor’s  60°  field-of-view.  In  addition  to  the  IR  imaging,  advanced  symbology  has 
been  developed  to  overlay  the  IR  image  to  provide  pilots  with  meaningful  cues  and  information 
required  for  landing.  Symbology  provides  an  artificial  representation  of  real  world  information 
and  can  serve  to  enhance  displays  (Curtis,  Jentsch,  and  Wise,  2010).  Properly  designed  visual 
displays  with  advanced  symbology  over  IR  images  are  intended  to  provide  pilots  with  increased 
situation  awareness  during  DVE.  However,  excessive  use  of  symbology  and  the  use  of 
undistinguishable  symbols  can  lead  to  clutter  and  confusion,  which  can  negatively  affect 
performance. 

Visual  symbology  sets  (Legacy  HUD  [AN/AVS-71,  BOSS,  and  FISH) 

The  Federal  Aviation  Administration  (FAA)  requires  an  altimeter,  airspeed  indicator,  and 
magnetic  compass  instruments  for  reference  during  visual  flight  rules  (VFR)  flights.  This 
indicates  that  even  with  good  visibility  and  a  discernable  horizon,  pilots  cannot  accurately  gauge 
airspeed,  altitude,  or  heading  without  the  aid  of  flight  instruments.  Additionally,  symbology 
provides  pilots  with  redundant  visual  cues. 

This  project  focused  on  three  visual  symbology  systems:  the  control  symbology  (Legacy 
HUD),  Brownout  Symbology  Set  (BOSS)  with  3-dimensional  (3D)  conformal  symbols,  and 
Forward-Looking  Integrated  Systems  for  Helicopters  (FISH).  For  evaluation,  each  of  these 
visual  symbology  systems  was  overlaid  on  an  IR  image  on  a  Head-Down  Display  (HDD).  A 
basic  overview  of  each  system  follows. 

Legacy 

The  Legacy  HUD  symbology  is  the  baseline  control  display  system  currently  used  in  many 
U.S.  Army  helicopters.  The  system’s  cues  include  attitude,  heading,  waypoint  bearing  and 
distance,  altitude,  performance,  and  velocity  vector.  The  system  was  designed  for  use  in 
conjunction  with  a  HUD  system;  however,  it  may  also  be  used  with  an  IR  scene  HDD. 

BOSS 

The  BOSS  provides  “visual  quality”  landing  capabilities  in  zero  visibility,  and  horizontal  and 
vertical  speed  guidance  to  the  landing  zone  (Thompson,  2011).  BOSS  utilizes  3D  conformal 
symbols,  which  allow  a  3D  view  of  the  landing  zone  added  to  the  two-dimensional  (2D) 
symbology  set.  Conformal  symbology  facilitates  the  mental  integration  of  information  outside 
the  helicopter  and  the  symbology  presented  on  the  display  (Wickens,  2003).  BOSS  was 
developed  using  symbology  tailored  for  rotorcraft  during  brownout  conditions  and  presents 
critical  flight  information  necessary  for  safe  takeoffs,  hovers,  and  landings  in  DVE.  A  2009 
study  of  BOSS  showed  that  pilots  were  able  to  effectively  use  the  symbology  to  land  helicopters 
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in  brownout  conditions;  however,  the  system  did  not  indicate  lateral  drift  at  a  level  considered  to 
be  suitable  for  safe  DVE  operations  (Szoboszlay,  Turpin,  and  McKinley). 

FISH 

FISH  uses  pictorial  pursuit  guidance  symbology  enroute,  and  prior  to  landing  switches  to 
forward-looking  landing  guidance  symbology.  FISH  symbology  maintains  a  forward  looking 
perspective,  which  is  consistent  with  how  pilots  naturally  fly  when  in  visual  flight  conditions. 
FISH  attempts  to  reduce  visual  search  times  by  integrating  all  cues  in  the  center  of  the  screen. 
Clutter  is  also  reduced  as  the  system  switches  from  an  Enroute  page  to  Hover  page  format  prior 
to  landing.  The  display  also  possesses  the  ability  to  reroute  the  trajectory  in-flight,  which  might 
be  needed  due  to  unforeseen  events  such  as  obstacles,  traffic,  or  an  unexpected  change  in  the 
direction  of  approach  to  the  landing  site.  In  both  simulator  and  in-flight  testing  of  FISH,  pilots 
were  able  to  achieve  improved  tracking  perfonnance,  fewer  missed  waypoints,  and  lower 
workload  (Moralez  et  ah,  2011). 

Visual  display  limitations 

Most  information  is  presented  visually  in  modern  cockpits;  thus,  the  visual  channel  can 
become  overloaded  while  operating  in  high-workload  conditions  such  as  DVE  (Veltman,  Oving, 
and  Bronkhorst,  2009).  Overreliance  on  any  one  sensory  channel,  especially  during  periods  of 
high  workload,  can  cause  cognitive  tunneling  and  sensory  bottleneck  (Allan  et  ah,  2010).  Visual 
channels  are  often  overburdened  by  cluttered  visual  displays  and  complex  symbology,  rendering 
pilots  susceptible  to  cognitive  tunneling.  Cognitive  tunneling  is  a  phenomenon  of  focusing  so 
intently  on  a  display  that  the  pilot  loses  focus  of  the  environment  as  a  whole.  As  more  visual 
attention  is  required,  the  visual  sense  may  become  overloaded  and  critical  information  may  be 
missed  or  misinterpreted  (Mateo  et  ah,  2012).  There  is  also  a  temporal  cost  to  cluttered  visual 
displays:  Displays  can  distract  or  slow  down  the  pilot  from  obtaining  necessary  infonnation. 
Longer  search  times  can  negatively  impact  perfonnance  and  increase  workload  (Curtis,  Jentsch, 
and  Wise,  2010).  Sensory  overload  associated  with  congested  displays  and  complex  symbology 
can  actually  cause  pilots  to  see  and  comprehend  less  as  more  infonnation  is  provided. 

Supplemental  cueing 


Aural 


In  DVE,  aural  alerts  can  be  beneficial  by  alerting  pilots  of  unintended  aircraft  orientations  and 
by  providing  pilots  with  navigational  reference  data,  i.e.,  in  the  form  of  altitude  callouts.  Aural 
alerts  are  most  effective  for  simple  short  messages  that  call  for  immediate  action  (Sanders  and 
McConnick,  1993).  Auditory  cues  possess  several  characteristics  that  can  make  them  preferable 
to  visual  cues,  particularly  when  visual  channels  are  already  overloaded.  Specifically,  auditory 
cues  have  the  ability  to  capture  a  pilot’s  attention  and  elicit  an  urgent  response  regardless  of  head 
position  or  eye  fixation  (Dehais  et  ah,  2014;  Vidulick,  Wickens,  Tsang,  and  Flach,  2010).  They 
are  especially  useful  when  operating  in  close  proximity  to  the  ground  where  unintentional  drift, 
changes  in  altitude,  and  sink  rates  require  immediate  corrective  action  to  prevent  a  potentially 
serious  accident.  Aural  alerts  can  be  speech  or  nonspeech  (tones/earcons)  (Kenny  and  Wei, 
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2009).  Earcons  are  a  form  of  nonspeech  notifications  composed  of  beeps  and  other  distinctive 
sounds  that  signal  distinct  events  and  are  used  extensively  in  computer  interfaces.  Nonspeech 
cues  such  as  earcons  have  potential  advantages  over  speech  in  their  ability  to  grab  attention  and 
be  concise,  specific,  and  distinctive.  However,  their  meaning  must  be  learned  and  retained.  The 
presentation  of  the  aural  alerts  can  range  from  monaural  to  3D  audio.  The  more  technically 
challenging  3D  audio  fonnat  allows  aural  alerts  to  be  localized  to  a  specific  location  in  space. 

Aural  cues  design  considerations 

Effective  aural  displays  rely  on  timely  and  accurate  interpretation  of  aural  alerts.  Overuse  of 
audio  displays  can  lead  to  auditory  clutter  of  competing  displays  and  increase  the  risk  for 
auditory  masking  (Curtis,  Jentsch,  and  Wise,  2010).  In  addition  to  being  masked,  research  has 
shown  that  during  periods  of  high  workload,  auditory  signals  are  often  missed.  For  example,  air 
safety  reports  show  that  a  significant  number  of  accidents  are  caused  by  lack  of  reaction  to 
auditory  alarms  due  to  inattentional  deafness,  a  phenomenon  that  occurs  when  high  perceptual 
load  tasks  consume  attentional  capacity  to  the  point  where  task-irrelevant  information  cannot  be 
processed  (  Dehais  et  al).  Auditory  displays  can  also  be  intrusive  and  distracting.  To  overcome 
these  limitations,  certain  design  guidelines  should  be  followed.  In  general,  signals  should  be 
discernable,  consistent,  and  not  provide  more  information  than  necessary.  Presentation  of  audio 
signals  should  avoid  extremes  of  auditory  dimensions,  establish  intensity  relative  to  ambient 
noise,  and  not  overload  the  auditory  channel.  Aural  cues,  while  suitable  for  grabbing  attention, 
are  not  recommended  for  long,  complex  messages  containing  information  that  will  need  to  be 
recalled  at  a  later  time. 

The  initial  aural  cues  are  designed  to  emulate  an  easily  fielded  system  that  has  already  been 
approved  for  flight  use.  To  this  end,  this  study  utilized  the  SwiftTalker  voice  audio  symbology 
system  to  monaurally  alert  pilots  of  altitude  and  unintentional  drifts.  Subsequent  studies  will 
incorporate  more  complex  aural  cues  (e.g.,  3D,  earcons).  SwiftTalker  verbal  alerts  are  provided 
through  aircrew  helmets.  SwiftTalker  uses  Text-to-Speech  (TTS)  technology  to  create  the  verbal 
alerts.  These  alerts  are  easy  to  manipulate,  nonreliant  on  human  participation,  have  a  low 
lifecycle  cost,  and  have  rapid  prototype-delivery  phases  and  iterations.  In  this  study,  SwiftTalker 
provided  pilots  with  altitude  alerts  during  approach  to  hover,  verbally  announcing  “250  FEET,” 
“40  FEET,”  “30  FEET,”  “20  FEET,”  and  “10  FEET.”  Altitude  alerts  were  linked  to  the  radar 
altimeter  to  ensure  accurate  feedback  was  provided  to  the  pilots.  Drift  detection  occurred  during 
the  30  foot  (ft.)  hover  condition  and  the  alerts  were  via  verbal  cueing.  If  a  lateral  drift  was 
detected,  the  system  announced  “DRIFTING  LEFT”  or  “DRIFTING  RIGHT,”  while 
longitudinal  drifts  were  announced  as  “DRIFTING  FORWARD”  or  “DRIFTING  AFT.” 
Additional  cues  alerted  pilots  when  to  accept  approach  guidance  via  the  voice  command 
“Assume  Guidance”  or  notified  pilots  if  their  heading  or  airspeed  did  not  match  desired 
parameters  via  the  voice  commands  “Check  Heading”  and  “Check  Speed.”  SwiftTalker  cues 
provided  pilots  with  altitude  information  that  was  redundant  with  all  symbology  sets. 

Tactile 


As  visual  and  aural  channels  have  become  overwhelmed  in  the  cockpit,  there  is  new  interest  in 
utilizing  the  sense  of  touch  with  tactile  cues  (Lu  et  al.,  2011).  A  primary  reason  for  utilizing 
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tactile  displays  is  that  their  use  results  in  minimal  interference  with  visual  and  aural  channels. 
This  is  largely  because  they  share  few  competing  attentional  resources.  Thus,  tactile  cues  can  be 
used  to  unburden  overtaxed  visual  and  aural  modalities  and  increase  overall  sensory  processing 
bandwidth.  The  increase  in  bandwidth  means  greater  attentional  capture  and  quicker  response 
times  for  infonnation  that  would  otherwise  be  delayed  or  missed  (Jones  and  Sarter,  2008). 
Experience  has  shown  that  tactile  cues  are  useful  in  providing  spatial  orientation  and  guidance 
information,  notifications  and  alerts,  and  flight-control  feedback  (Allan,  et  ah,  2010).  Tactile 
cues  are  especially  useful  in  demanding  and  distracting  environments  and  in  some  cases  they 
have  outperformed  visual  displays  under  conditions  of  high  cognitive  and  visual  workload 
(Elliott,  Schmeisser,  and  Redden,  2011). 

Tactile  displays  usually  convey  information  using  vibrations  or  force.  In  the  cockpit,  tactile 
cues  can  be  integrated  in  the  flight  controls  or  may  be  delivered  through  a  vest,  belt,  and/or  seat 
apparatus.  In  practice,  arrays  of  electromagnetic  vibro-tactile  stimulators  called  tactors  are  used 
to  create  the  tactile  display  (Lawson  and  Rupert,  2014).  Currently,  tactors  cue  pilots  via  a  unique 
tapping  sensation  on  the  shoulders,  around  the  waist,  and  in  the  seat  cushion  (Cox,  2014).  For 
example,  directional  flight  deviations  can  be  cued  on  a  corresponding  belt  location,  with  an 
unintended  drift  to  the  right  being  cued  on  the  right  side  of  the  belt  and  the  rate  of  deviation 
signaled  by  tactor  firing  frequency  rate.  Systems  utilizing  this  encoding  scheme  have 
demonstrated  improved  system  awareness  and  heightened  situational  awareness.  Key  design 
considerations  include  the  tactor  size  and  strength,  user  comfort,  ease  of  donning,  and 
anthropomorphic  considerations  (bulk,  egress,  wearability,  and  flight  gear  compatibility). 

Tactile  system  selection 

Tactile  cueing  was  selected  based  on  its  potential  ability  to  aid  pilots  operating  in  DVE; 
specifically,  tactile  cueing  was  used  to  provide  redundant  drift,  course,  and  altitude  information. 
The  tasks  pilots  performed  (approach  to  landing,  approach  to  hover,  hover,  and  sidestep)  were 
selected  because  the  test  a  pilot’s  ability  to  enter  and  operate  in  DVE  conditions.  Additionally,  it 
was  necessary  that  the  tactile  display  ensemble  be  comfortable,  compatible  with  aircraft 
operation,  and  allowed  for  emergency  egress.  Finally,  the  tactile  display  system’s  software  and 
hardware  was  integrated  into  USAARL’s  NUH-60FS. 

The  Tactile  Situation  Awareness  System  (TSAS)  was  found  to  meet  all  selection 
requirements.  Specifically,  it  used  noncontinuous  tactor  stimulation  to  preclude  sensory 
habituation,  met  airworthiness  requirements,  and  was  compatible  with  the  NUH-60  simulator 
software  requirements.  In  flight  tests,  TSAS  was  shown  to  be  capable  of  providing  altitude, 
attitude,  velocity,  navigation,  acceleration,  threat  location,  and  target  location  data  (McGrath  et 
ah,  2004).  Additionally,  TSAS  has  shown  to  be  effective  at  reducing  tracking  errors  and 
improving  situational  awareness  during  landings  in  both  degraded  and  good  visual  environments 
(Craig  et  al.,  2008).  Further,  Kelley,  Grandizio,  Estrada,  and  Crowley  (2014)  found  that  aviator 
performance  with  vibro-tactile  displays  was  not  adversely  affected  by  adaptation  or  habituation 
following  12  continuous  hours  of  simulated  flight.  The  TSAS  system  selected  and  used  in  this 
experiment  consisted  of  eight  tactors  along  the  belt  that  correspond  to  direction  of  drift  or  course 
deviation.  Shoulder  and  seat  tactors  reported  altitude  deviations.  On  course  forward  flight  was 
cued  via  the  center  tactor. 
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Multisensory  approach 


Flying  in  DVE  presents  pilots  with  the  potential  for  high  workload  and  sensory  overload. 
Single  modality  solutions  can  increase  the  already  high  workload  and  provide  an  incomplete 
picture  of  the  outside  world,  resulting  in  a  negative  effect  on  perfonnance.  Sensory  overload  and 
increased  workload  can  lead  to  missed  cues,  loss  of  situation  awareness  (SA),  and  adversely 
affect  overall  safety.  To  overcome  the  risks  associated  with  DVE,  effective  and  efficient  use  of 
pilot  resources  are  required.  In  recent  years,  interest  in  multimodal  interfaces  has  increased  for 
complex,  event-driven  domains  that  are  at  risk  for  sensory  overload  due  to  an  overreliance  of 
visual  displays  (Sarter,  2007).  Wickens’  Multiple  Resource  Theory  (MRT)  predicts  that 
performance  can  be  improved  by  distributing  infonnation  across  sensory  channels. 

According  to  MRT,  humans  are  capable  of  processing  information  from  multiple  sensory 
sources  in  parallel.  Thus,  pilots  are  capable  of  processing  visual,  sound,  and  tactile  inputs 
simultaneously  using  multiple  sensory  resources  (Vidulick,  Wickens,  Tsang,  and  Flach,  2010). 
Tasks  using  compatible  resources  that  allow  parallel  processing  may  usually  be  performed 
simultaneously.  Multimodal  systems  support  time-sharing  and  attention  management.  Based  on 
MRT,  a  multimodal  approach  that  utilizes  visual,  audio,  and  tactile  senses  may  provide  pilots 
with  the  information  required  for  safe  DVE  operations  and  prevent  overreliance  on  the  visual 
sense.  Many  bimodal  research  studies  in  which  auditory  and  tactile  cues  have  been  introduced  to 
provide  directional  and  navigational  guidance  have  supported  this  theory.  A  meta-analysis  of 
more  than  600  studies  investigated  the  effectiveness  of  tactile  cues  versus  visual  cues  versus 
visual-tactile  cues  and  found  that  a  multisensory  approach  using  complementary  visual  and 
tactile  cues  increased  performance  for  orientation,  task  information,  and  alerts  (Elliott  et  ah, 
2009).  Research  conducted  by  Sklar  and  Sarter  (1999)  investigated  response  times  for 
uncommanded  changes  of  an  automatic  flight  deck  system  using  tactile,  visual,  and  tactile- 
visual  cueing.  It  was  found  that  response  times  for  tactile  and  tactile-visual  conditions  were 
significantly  better  than  response  times  for  a  visual  only  condition,  demonstrating  the  advantage 
to  bimodal  presentations.  Research  that  explored  the  efficacy  of  audio-tactile  systems  found  that 
well-designed  audio-tactile  displays  have  the  potential  to  result  in  more  resilient  systems  that 
enable  the  operator  to  receive  the  necessary  infonnation,  even  when  one  modality  is 
compromised  (Mateo  et  ah,  2012). 

Possible  limitations  to  multimodal  approaches 

While  there  is  great  promise  with  a  multisensory  approach  to  solving  the  DVE  problem,  it  is 
important  to  consider  certain  limitations  of  the  multimodal  approach.  Moving  from  uni-  or  bi-  to 
multimodal  displays  involves  certain  tradeoffs.  Multimodal  systems  may  aid  in  time-sharing,  but 
there  is  also  a  potential  increase  in  interface  management  and  monitoring  demands.  Due  to 
limited  capabilities  in  regard  to  human  infonnation  processing,  multisensory  cueing  could 
overload  the  pilots’  cognitive  abilities,  resulting  in  increased  workload  and  missed  cues. 

An  additional  cautionary  note  to  consider  is  The  Principle  of  Inverse  Effectiveness  in 
Multisensory  Integration  which  states: 
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As  the  strength  of  multisensory  integration  responses  increase,  the  strength 
of  responsiveness  to  individual  sensory  stimuli  decreases.  Consequently, 
multisensory  cueing  indices  will  naturally  serve  to  improve  associated 
performance  when  compared  to  individual  stimuli.  This  improved  degree  of 
performance  may  be  illusory  to  a  certain  degree,  merely  by  the  nature  of  the 
multisensory  inputs  (Holmes,  2009). 

Studies  are  needed  to  determine  if  tactile,  audio,  and  visual  cues  are  effective  at  directing 
pilots  in  DVE  and  to  establish  which  modes  are  most  effective  in  delivering  certain  types  of 
information.  Another  consideration  which  should  be  addressed  is  whether  audio  and  tactile  cues 
should  merely  provide  pilots  with  redundancy  for  missed  visual  cues  or  if  they  should  provide 
additional  information  that  visual  displays  are  not  capable  of  effectively  providing. 

Study  rationale  and  expectations 

There  is  a  multitude  of  research  exploring  bimodal  systems,  but  trimodal  systems  using  visual, 
aural,  and  tactile  cueing  research  is  limited.  It  was  the  goal  of  this  study  to  determine:  1)  if 
combining  symbology/cueing  sets  would  improve  flight  performance  and/or  reduce 
workload/stress,  2)  if  the  effectiveness  of  different  combinations  of  the  symbology/cueing  sets 
would  be  reflected  in  their  subjective  evaluations,  observed  flight  perfonnance,  and  pilot 
workload/stress,  and  3)  if  the  effectiveness  of  different  combinations  of  symbology/cueing  sets 
would  vary  with  the  flight  task. 

Study  metrics 

The  compatibility  and  effectiveness  of  each  combination  of  the  sensed  IR  scene,  visual 
symbology,  and  sensory  cueing  were  assessed  with  quantitative  measures  of  flight  performance 
collected  from  the  simulator,  biometrics  collected  from  the  pilot,  and  the  pilot’s  subjective 
reports.  Symbology  set  effectiveness  was  evaluated  three  ways:  1)  the  flight  performance 
enabled,  2)  the  degree  of  workload  and  stress  induced,  and  3)  the  evaluation  pilots’  assessments 
evoked.  Simulator  data  documented  the  symbology  sets’  effect  on  flight  perfonnance. 
Biometrics  (i.e.,  heart  rate  variability,  respiratory  rate,  and  galvanic  skin  response)  were 
collected  as  measures  of  the  symbology  sets’  effect  on  workload  and  stress.  Cooper-Harper, 
Bedford  Workload  Scale,  and  Visual  Cue  Index  (VCI)  data  along  with  free  reports  documented 
the  evaluation  pilots’  assessments  of  the  symbology/cueing  sets’  utility. 

Deliverables 


The  deliverables  of  this  effort  were  to: 

1 .  Determine  the  relative  efficacy  of  the  specified  visual  symbologies  when  teamed  with 
tactile  and  aural  cues; 

2.  Evaluate  the  effect  of  these  multiple  simultaneous  cueing  technologies  and  their  effect 
on  flight  performance,  biometrics,  workload,  and  situational  awareness;  and 

3 .  Provide  recommendations  for  managing  the  integration  of  these  advanced  cueing 
technologies. 
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Method/Test  procedures 
Test  equipment 


Flight  simulator 

All  testing  was  conducted  using  USAARL’s  NUH-60FS  research  flight  simulator. 
Imagery/Cue  display 

IR  scene  and  symbology  infonnation  was  shown  on  the  primary  flight  display  of  the  UH-60M 
instrument  panel  emulation.  Tactile  cues  were  presented  via  TSAS  belt,  shoulder  harness,  and 
seat  cushion  tactors.  Audio  cues  were  presented  via  speakers  inside  HGU-56/P  rotary-wing 
aircrew  helmets.  The  windscreen  and  chin  bubble  exterior  views  were  clouded  with  realistic 
dust  and  obscured  to  match  the  aircraft  integration  test  configuration  currently  utilized  by 
Aeroflightdynamics  Directorate,  National  Aeronautics  and  Space  Administration  (NASA)-Ames 
Research  Center. 

Instrumentation  of  evaluation  pilots 

The  test  instrumentation  used  by  the  evaluation  pilots  included:  HGU-56/P  rotary-wing 
aircrew  helmets;  TSAS  tactor  belt,  shoulder  harness,  and  seat  cushion;  and  biometric 
instrumentation  consisting  of  three  electrocardiogram  (ECG)  electrodes,  two  galvanic  skin 
response  (GSR)  electrodes,  a  chest  strap  respiration  transducer,  and  BioNomadix&  wireless 
transmitter. 

Flight  tasks  and  standards 

The  sponsor  requested  that  the  display  be  evaluated  with  flight  tasks  derived  from  ADS-33. 
Table  1  lists  the  four  derived  flight  tasks  selected.  The  flight  tasks  were  flown  in  the  order 
presented  for  each  symbology/cueing  combination. 

Table  1. 

Flight  tasks  derived  from  ADS-33. 


Flij 

ght  Task  Order 

First 

Approach/Landing 

Second 

Approach/Hover 

Third 

Hover 

Fourth 

Sidestep 

Approach  to  landing 

This  task  started  with  the  aircraft  250  ft.  above  ground  level  (AGL)  moving  at  80  knots  toward 
the  landing  point  1.5  nautical  miles  (NM)  away.  Descent  from  250  ft.  AGL  began  0.8  NM  from 
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the  landing  point.  The  pilots  approached  the  landing  point  in  a  straight  line  and  touched  down 
with  1 .5  to  2  knots  ground  speed  and  minimal  hover.  Metrics  for  this  task  included  deviations 
from  an  ideal  approach  path,  touchdown  speed,  touchdown  heading,  and  touchdown  location. 

Approach  to  hover 

This  task  started  with  the  aircraft  250  ft.  AGL  moving  at  80  knots  toward  the  landing  point  1.5 
NM  away.  Descent  from  250  ft.  AGL  started  0.8  NM  from  the  hover  point.  The  pilots 
approached  the  hover  point  in  a  straight  line  and  established  a  30  ft.  AGL  hover.  Metrics  for  this 
task  included  deviations  from  an  ideal  approach  path  and  hover  quality  (heading,  altitude,  and 
position). 

Hover 

This  task  required  the  maintenance  of  a  30  ft.  AGL  hover  for  2  min.  Metrics  for  this  task 
included  deviations  from  an  ideal  hover  quality  (heading,  altitude,  and  position). 

Sidestep 

From  a  hover,  the  pilot  rapidly  relocated  the  aircraft  using  a  sidestep  maneuver  and  returned  to 
a  stable  hover  above  a  predesignated  spot.  Metrics  included  maximum  lateral  velocity,  altitude 
maintenance,  heading  maintenance,  relocation  accuracy,  and  20  s  pre-  and  20  s  post-hover 
quality  (heading,  altitude,  and  position).  Crashes,  loss  of  control,  missed  approaches,  and/or 
aborted  landings  were  reported  separately. 

Biological  metrics 

The  projected  effect  of  the  flight  instrumentation  on  perceived  workload  utilized  three 
biometric  measures:  heart  rate  variability  (HRV),  galvanic  skin  response  (GSR),  and  respiratory 
rate  (RR).  These  measures  were  selected  for  their  responsiveness  to  workload,  minimal 
invasiveness,  compatibility  with  the  testing  paradigm,  tolerance  of  the  simulator  environment, 
and  potential  utilization  in  aircraft.  The  biometric  data  were  synchronously  collected  with 
BIOPAC’s  BioNomadixR  and  BHAPI  instrumentation/software.  This  instrumentation,  like  the 
biometric  measures,  was  selected  for  its  tolerance  of  the  simulator  testing  environment  and 
potential  utilization  in  aircraft.  Each  specific  biometric  data  array  were  analyzed  with 
BIOPAC’s  AcqKnowledge R  and  MathWorks’ ®  MATLAB !  software. 

HRV  has  been  used  successfully  to  observe  quick  or  minute  shifts  in  documented  workload 
(interpreted  in  the  current  study  as  stress)  that  have  eluded  simpler  measures  of  heart  rate.  HRV 
data  capture  requires  high  resolution  ECG  instrumentation  that  reports  detailed  heart  activity  data 
(specifically  the  R-R  interval).  The  selected  high  resolution  (500+  Hertz)  ECG  instrumentation 
was  easily  worn,  and  reported  wirelessly  as  part  of  the  BioNomadix  8  system  (although  applied 
successfully  in  the  NUH-60FS  simulator,  any  planned  or  anticipated  use  of  this  instrumentation 
in  the  UH-60  aircraft  would  require  an  approved  Airworthiness  Certification). 
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GSR  has  proven  effective  at  discerning  cognitive  workload  changes  in  naval  simulator  studies 
and  demonstrated  greater  sensitivity  to  shifts  in  workload  than  traditional  subjective  or  subject- 
reported  measures.  GSR,  also  commonly  referred  to  as  skin  conductance  or  electrodermal 
activity,  involves  the  measure  of  electric  conductivity  alterations  caused  by  the  skin’s  ionic  sweat 
production.  The  means  of  collecting  this  information  is  noninvasive;  the  selected  equipment  is 
compact,  created  little  or  no  discomfort  to  the  test  pilots,  and  reported  wirelessly.  Through  the 
BIOP  AC  hardware,  GSR  was  integrated  into  the  multichannel  ECG  array  utilized  for  HRV  data 
collection. 

RR  has  been  used  successfully  as  a  measure  of  cognitive  workload  and  stress  in  the  literature, 
including  multitasking  driving  simulator  research,  which  suggested  appropriateness  for  flight 
simulator  studies  (ignoring  any  possible  6  degrees  of  freedom  initiated  variability).  Respiratory 
rate  describes  the  rate  of  breathing,  and  in  some  cases  can  also  include  the  volume  of  a  test 
pilot4 s  inhalation  via  measured  lung  expansion.  Respiratory  rate  data  can  be  either  extracted 
from  high-resolution  ECG  data  or  independently  collected  through  a  respiration  transducer. 

While  the  independent  RR  collection  method  requires  a  chest  strap,  it  was  selected  because  of  its 
increased  signal-to-noise  ratio,  which  allowed  for  a  wider  array  of  analysis  (e.g.,  inclusion  of 
breath  volume  as  opposed  to  simple  breathing  rate  data).  Through  the  BIOP  AC®  hardware,  RR 
was  integrated  into  the  multichannel  ECG  array  utilized  for  heart  rate  variability  and  GSR  data 
collection. 

Qualitative  subjective  metrics 


Following  completion  of  each  scored  symbology  set  combination  test  run,  pilot  subjective 
impressions  were  captured  using  the  Cooper-Harper  Handling  Qualities  Rating  Scale,  Bedford 
Workload  Scale,  and  VCI  ratings.  The  Visual  Cue  paradigm  was  used  to  rate  aural  and  tactile 
cues  in  addition  to  visual  symbology.  After  test  pilots  finished  all  flight  combinations,  they  were 
asked  to  complete  a  questionnaire,  ranking  cueing  displays  from  easiest  to  fly  -  to  most  difficult, 
for  each  symbology  set.  The  questionnaire  also  captured  flight  experience  and  free  report 
sections.  During  the  testing,  research  pilots  collected  test  pilot  free  flow  comments. 

Test  conditions 


Test  flights  were  flown  with  a  single  unassisted  (minimal  crew  coordination)  evaluation  pilot 
at  the  controls  with  wind  and  turbulence  turned  off.  The  out-the -window  views,  including  the 
chin  bubbles,  were  obscured  with  blowing  sand  and  dust  below  100  feet  AGL  for  all  flights.  The 
IR  scene  imagery  within  the  display  was  unobscured. 

Test  order 


Training  and  testing  required  two  days  for  each  of  the  eight  evaluation  pilots.  The  first  day 
began  with  a  safety  brief,  description  of  the  study’s  purpose  and  instrumentation,  and  six 
simulator  flight  hours  of  symbology  socialization/training.  On  the  second  day,  test  runs  (each 
run  is  composed  of  the  four  selected  flight  tasks:  approach  to  landing,  approach  to  hover,  hover, 
and  sidestep)  were  flown.  Three  consecutive  test  runs  were  flown  for  each  of  the  12 
symbology/cueing  combinations  shown  in  table  2.  The  first  two  runs  provided  additional 
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training  and  preparation  for  the  third  run,  which  was  for  score.  Thus,  on  the  second  day,  each 
pilot  practiced  each  of  the  12  symbology  combinations  listed  in  table  2  twice  and  then  once  for 
score,  for  a  total  of  36  runs,  12  of  which  were  scored  runs.  In  turn,  each  of  these  12  scored  runs 
provided  4  data  sets:  approach  to  landing,  approach  to  hover,  hover,  and  sidestep.  Thus,  48  data 
sets  were  harvested  from  each  of  the  eight  evaluation  pilots.  The  pseudorandom  testing 
sequence  is  shown  in  table  3. 


Table  2. 

Symbology/cueing  combinations  numbering  system. 


Visual  Symbology  Set 

IR  Scene 

IR  Scene  + 
Tactile 

IR  Scene  + 
Aural 

IR  Scene 
+ 

Tactile 

+ 

Aural 

Legacy 

1 

2 

Q 

4 

BOSS  +  3D  Conformal 

5 

6 

7 

8 

FISH 

9 

10 

11 

12 

Table  3. 

Pseudorandomized  test  order  of  symbology  combinations. 


Runs 

1 

2 

4 

Test  pilot 
4  5 

6 

7 

8 

1,2,3  k 

3 

8 

A 

2 

7 

8 

10 

1 

1  4,5,6 

>  4 

12 

© 

3 

11 

2 

6 

9 

7,  s,  9  y 

8 

4 

V 

12 

3 

10 

1 

12 

10, 11, 12 

1 

3 

11 

9 

9 

4 

12 

5 

13, 14, 15 

5 

7 

5 

8 

5 

1 

9 

10 

16, 17, 18 

10 

1 

1 

11 

8 

7 

8 

8 

19,  20,21 

7 

5 

2 

10 

1 

12 

4 

11 

22,  23,  24 

2 

11 

8 

4 

10 

3 

3 

7 

25,  26,  27 

9 

2 

10 

6 

6 

11 

5 

2 

28,  29,  30 

11 

9 

4 

1 

12 

5 

7 

4 

31,32,  33 

6 

6 

7 

5 

4 

9 

11 

6 

34,  35,  36 

12 

10 

9 

7 

2 

6 

2 

3 

As  seen  in  table  3,  the  pseudorandomized  order  of  the  symbology/cueing  combinations  (from 
table  2)  was  specified.  For  example,  test  pilot  3’s  fourth,  fifth,  and  sixth  runs  were  test 
symbology  combination  no.  3  (circled).  As  shown  in  table  2,  symbology  combination  no.  3 
utilizes  aural  cues  and  has  the  IR  scene  overlaid  with  AN/AVS-7  Legacy  FIUD  symbology. 

Simulator  time  requirements 

The  study  was  designed  to  enable  a  pilot  to  complete  data  collection  in  2  days.  The  first  day 
was  dedicated  to  socialization  of  the  visual  symbology,  tactile  cues,  and  aural  cues  and  took 


12 


roughly  6  hours  per  pilot.  The  second  day  was  dedicated  to  the  test  runs.  Each  pilot  flew  6.6 
hours  of  flight  time  (roughly  1 1  minutes  for  each  of  the  36  test  runs). 

Evaluation  pilots 

The  test  plan  was  specifically  designed  to  provide  a  structured  environment  to  capture  expert 
pilot  flight  performance  and  subjective  assessments  of  display,  symbology,  and  cueing 
characteristics.  Accordingly,  the  sponsor  selected  the  eight  formally  trained  and  rated  rotary¬ 
wing  pilots  who  served  as  the  evaluation  pilots.  The  evaluation  pilots  had  sufficient  flight 
experience  to  enable  them  to  provide  expert  guidance  in  the  establishment  of  display 
requirements.  The  evaluation  pilots  perfonned  the  flight  tasks  and  provided  subjective 
estimations  of  workload  and  cue  utility,  biometric  measures  of  workload  and  stress,  and 
objective  measures  of  their  flight  performance.  Demographics  of  the  evaluation  pilots’  flight 
experiences  were  collected  with  a  questionnaire. 

Analysis 

Flight  performance  data  were  evaluated  for  pilot  perfonnance  for  approach  to  landing, 
approach  to  hover,  hover,  and  sidestep  maneuvers.  Data  were  captured  on  the  third  run  for  each 
condition.  Repeated  measures  Analysis  of  Variance  (ANOVA)  was  performed  on  position, 
altitude,  heading,  and  speed  measures.  A  logarithmic  transfonnation  of  the  data  was  performed 
to  pass  the  Shapiro-Wilks  test  for  normality  required  of  most  performance-specific  statistical 
testing.  Means  were  reported  using  raw  data,  while  statistical  significance  was  derived  from  log- 
transformed  data.  A  p  value  less  than  .05  was  used  for  statistical  significance.  Flight 
performance  data  were  grouped  by  maneuver  and  displayed  as  a  column  graph.  Perfonnance 
measure  indices  are  grouped  by  symbology.  Each  symbology  bar  is  the  average  of  all  four 
combinations  of  cueing  displays  (IR,  TSAS,  Aural,  TSAS  and  Aural).  Cueing  graphs  display 
results  for  a  specific  performance  measure  used  to  compute  the  perfonnance  index  within  a 
specific  symbology  set.  Only  charts  and  data  with  statistically  significant  findings  are  presented. 
Tables  showing  both  statistically  significant  and  non-significant  results  are  located  in  appendix  a. 

Subjective  reports  include  results  by  maneuver  for  Cooper-Harper,  Bedford  Workload  Scale, 
and  VCI  ratings.  Friedman  Tests  were  performed  on  the  subjective  results  to  test  for  differences 
between  the  12  conditions.  Further  investigations  with  a  Wilcoxon  Sign  Rank  test  were 
performed  between  visual  symbologies  and  cueing  conditions. 

Pilot  questionnaire  data  (appendix  b),  which  required  pilot  assessment  of  cueing  within 
symbologies,  were  then  examined  and  presented  as  rank  data.  Raw  data  are  represented  to 
indicate  pilot  preference.  Due  to  the  confusing  nature  of  the  original  questionnaire  given  to 
pilots  1  and  2,  a  revised  questionnaire  was  used  for  rank  data  and  only  the  data  from  pilots  3 
through  8  were  analyzed.  Finally,  pilot  biometric  data  analysis  includes  summaries  for  both 
condition  and  maneuver.  In  the  following  sections,  data  for  each  task  are  presented  graphically 
and  evaluated  statistically. 
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Results 


Approach  to  landing 

To  determine  the  optimized  cueing  display  configurations  used  to  facilitate  helicopter 
approach  to  landing  in  DVE,  all  combinations  of  cueing  were  recorded  and  analyzed.  Result 
tables  are  provided  for  both  symbology  comparisons  and  cueing  comparisons.  Mean  deviations 
and  statistical  significance  for  each  measure  are  shown  in  tables  4  and  5. 

Flight  performance 

In  the  approach  to  landing  maneuver,  statistical  significance  between  symbologies  was  found 
for  the  following  measures:  position,  heading,  altitude,  and  speed  (table  4).  Cueing 
effectiveness  was  detennined  using  pairwise  comparisons  within  a  symbology  set  using  the 
components  of  the  indices.  Table  5  shows  cueing  significance  for  BOSS  for  position  and  for 
Legacy  for  heading  and  speed. 


Table  4. 

Mean  deviations  by  symbology  and  significance  values  between  symbologies. 


Approach  to 
landing 

Mean 

/;-value  (p  <  .05) 

Index 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Position  (feet) 

22.683 

9.823 

15.465 

<0.001 

0.075 

0.151 

Heading  (deg.) 

3.656 

2.661 

3.900 

0.001 

0.960 

<0.001 

Altitude  (feet) 

9.849 

8.816 

7.967 

0.270 

0.007 

0.381 

Speed  (knots) 

9.946 

6.703 

8.184 

0.003 

0.271 

0.225 

14 


Table  5. 

Cueing  mean  deviations  and  significance  values  for  approach  to  landing. 


Approach 
to  landing 

Mean  deviation 

/;-value  (p  <  .05) 

Index 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Position 

(ft.) 

BOSS 

10.27 

8.163 

6.941 

13.92 

0.03 

0.048 

0.347 

0.734 

0.439 

0.443 

Heading 

(deg.) 

Legacy 

(Level 

phase) 

0.75 

0.543 

0.888 

0.353 

0.229 

0.672 

0.144 

0.183 

0.949 

0.045 

Speed 

(kts) 

Legacy 

(Level 

phase) 

0.391 

0.586 

1.361 

0.829 

0.522 

0.017 

0.404 

0.096 

0.607 

0.054 

Legacy 

(Landing) 

3.234 

2.655 

3.896 

2.551 

0.279 

0.538 

2.15 

0.048 

0.961 

0.237 

Legacy 

(Landing 

Lateral 

Drift) 

0.322 

0.414 

0.161 

0.127 

0.375 

0.07 

0.046 

0.048 

0.06 

0.305 

*  “Both”  re 


ers  to  the  combined  TSAS  and  Aural  display. 


Position 


The  position  index  indicates  pilot  deviations  from  the  ideal  point  in  the  touchdown  zone.  The 
ideal  point  could  be  seen  visually  by  aligning  the  aircraft  in  the  center  of  a  Y-light  with  the 
forward  light  straight  ahead  of  the  helicopter  and  side  Y-lights  would  ideally  be  centered  off  the 
side  windows  when  the  dust  cleared.  BOSS  and  FISH  symbologies  provided  guidance  to  the 
ideal  touchdown  point,  whereas  with  the  Legacy  symbology,  pilots  were  forced  to  detennine 
position  using  the  IR  sensor  alone.  The  measurement  for  analysis  was  the  position  where  all 
three  wheels  of  the  aircraft  first  touched  down  and  not  the  final  resting  position  after  the  aircraft 
came  to  a  stop. 
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Approach  to  Landing 
Position  Deviation 
Symbology 


Legacy  BOSS  FISH 

Error  Bars  +/- 1  S.E. 


Approach  to  Landing 
Position  at  Touchdown 
BOSS  Symbology 
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IR  TSAS  Aural  Both 


Error  Bars  +/- 1  S.E. 


Figure  2.  Position  index  data  by  symbology  (left)  and  position  at  touchdown  data  for  BOSS  by 
cueing  (right).  Matching  data  labels  indicate  which  symbologies/cueings  are 
statistically  significantly  different.  Note:  “Both”  refers  to  the  combined  TSAS  and 
Aural  display. 

Symbology.  Flight  performance  for  the  position  index  (figure  2  left)  showed  that  BOSS  had 
the  lowest  mean  deviation  and  pilot  performance  was  significantly  better  with  BOSS  than  with 
the  Legacy  symbology. 

Cueing.  Figure  2  (right)  shows  significant  differences  in  touchdown  position  for  cueing 
within  the  BOSS  symbology.  The  data  indicate  that  pilot  position  accuracy  improved  when 
BOSS  was  paired  with  either  the  TSAS  or  Aural  cueing  displays  than  when  using  BOSS  paired 
with  only  an  IR  scene. 

Heading 

During  all  portions  of  the  approach  to  landing,  pilots  were  told  to  maintain  a  040  degree 
heading.  The  performance  measures  used  to  compute  the  heading  deviation  index  were  the 
standard  deviation  (SD)  of  the  heading  during  the  level,  transition  to  approach,  approach,  and 
transition  to  landing  segments.  Heading  was  not  recorded  after  the  landing  was  completed. 
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Approach  to  Landing 
Heading  Deviation 
Symbology 


Legacy  BOSS  FISH 

Error  Bars  +/- 1  S.E. 


Approach  to  Landing  Level 
Segment  SD  Heading 
_  Legacy  Symbology 

Sf  1.2  -i - * - 


■o 


IR  TSAS  Aural  Both 

Error  Bars  +/- 1  S.E. 


Figure  3.  Heading  index  data  by  symbology  (left)  and  the  SD  of  heading  during  the  level 

segment  of  the  maneuver  for  Legacy  by  cueing  (right).  Note:  “Both”  refers  to  the 
combined  TSAS  and  Aural  display. 

Symbology.  Figure  3  (left)  shows  flight  performance  for  the  heading  deviation  index.  The 
data  indicate  that  pilots  performed  better  while  using  BOSS  symbology  for  overall  heading 
maintenance.  Pilot  performance  was  significantly  better  with  BOSS  than  with  either  FISH  or 
Legacy  symbologies. 

Cueing.  Figure  3  (right)  shows  within  symbology,  heading  performance  was  affected  by 
cueing  in  the  Legacy  symbology  set  during  the  level  segment  of  the  maneuver.  Pilot  heading 
maintenance  was  significantly  better  with  the  TSAS  and  Aural  cueing  display  than  with  the 
Aural  cueing  display. 

Altitude 


The  only  performance  measure  used  for  the  altitude  deviation  index  was  the  root  mean  square 
deviation  (RMSD)  of  the  altitude  during  the  level  segment  where  pilots  were  required  to 
maintain  250  ft.  of  altitude.  During  the  rest  of  the  maneuver,  the  BOSS  symbology  did  not 
provide  data  on  altitude,  but  rather  a  velocity  vector,  so  it  was  not  comparable  to  the  FISH  and 
Legacy  symbologies. 
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Figure  4.  Altitude  index  data  by  symbology. 

Symbology.  Figure  4  shows  that  flight  performance  for  altitude  deviation  during  the  level 
portion  was  lowest  with  FISH  symbology.  Statistical  significance  was  found  between  FISH  and 
Legacy  symbologies. 

Cueing.  No  significant  difference  in  altitude  deviation  was  found  among  cueing  displays. 
Speed 

The  performance  measures  used  to  calculate  the  speed  deviation  index  were  the  RMSD  of  the 
speeds  during  the  level  and  approach  segments  (80  knots),  and  the  difference  between  the 
measured  speeds  and  the  target  speeds  in  the  longitudinal  and  lateral  directions  at  the  moment  of 
touchdown. 
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Figure  5.  Speed  index  data  by  symbology  (left)  and  the  RMSD  of  speed  during  the  level 

segment  for  Legacy  by  cueing  (right).  Note:  “Both”  refers  to  the  combined  TSAS  and 
Aural  display. 

Symbology.  Figure  5  (left)  shows  flight  performance  for  the  speed  index.  Overall,  speed 
performance  was  best  when  pilots  were  using  advanced  symbology.  BOSS  had  the  lowest  mean 
deviation  for  landing  speed  performance.  Statistical  analyses  indicated  that  performance  was 
significantly  better  with  BOSS  than  with  Legacy  symbology. 

Cueing.  Figure  5  (right)  shows  within  symbology,  cueing  effects  were  seen  in  both  the  level 
and  landing  phases.  In  the  level  phase,  pilot  deviation  from  the  80  knot  required  speed  was  less 
with  Legacy  display  (.4  knots)  than  with  the  Legacy  and  Aural  cueing  display  (1.4  knots,/?  = 
.017). 


19 


Approach  to  Landing 
Longitudinal  Speed  at  Touchdown 
Legacy  Symbology 


IR  TSAS  Aural  Both 

Error  Bars  +/- 1  S.E. 


Approach  to  Landing 
Lateral  Speed  at  Touchdown 
Legacy  Symbology 

0.6 
0.5 
T?  0.4 
c  0.3 

TO 

a) 

S  0.2 
0.1 
0.0 

IR  TSAS  Aural  Both 

Error  Bars  +/- 1  S.E. 


Figure  6.  Longitudinal  speed  at  touchdown  data  (left)  and  lateral  speed  at  touchdown  data  (right) 
for  Legacy  condition  and  grouped  by  cueing.  Note:  “Both”  refers  to  the  combined 
TSAS  and  Aural  display. 

Figure  6  (left)  shows  cueing  significance  for  the  touchdown  phase  of  the  approach  to  land 
maneuver  was  found  for  the  Legacy  symbology  set.  Pilots  were  directed  to  touch  down  with  1  to 
2  knots  of  forward  (longitudinal)  airspeed.  The  data  suggest  that  forward  landing  speed  was 
significantly  slower  with  tactile  cueing  than  with  the  Aural  cueing  display  when  flying  the 
Legacy  symbology. 

Figure  6  (right)  shows  in  the  landing  phase,  pilots  were  directed  to  touch  down  with  no  lateral 
drift.  Drift  performance  differences  existed  for  Legacy  symbology.  Drift  performance  was  best 
when  Legacy  symbology  was  paired  with  the  TSAS  and  Aural  cues  display.  Drift  performance 
was  statistically  better  when  Legacy  was  paired  with  the  TSAS  and  Aural  cueing  display  than 
when  paired  with  only  the  IR  scene.  Additionally,  perfonnance  using  the  Aural  cueing  display 
was  significantly  better  than  performance  using  the  TSAS  cueing  display. 

Subjective  data 

Following  the  approach  to  landing  maneuver,  pilots  were  asked  to  rate  the  effect  of  cueing 
displays  on  handling  qualities  using  the  Cooper-Harper  Handling  Qualities  Rating  Scale, 
workload  using  the  Bedford  Workload  Scale,  and  usable  cue  index  using  the  VCI.  Table  6 
shows  that  overall  ratings  indicated  pilot  preference  for  BOSS  symbology  over  Legacy 
symbology.  Bedford  Workload  Scale  data  indicate  that  pilots  perceived  workload  to  be 
significantly  lower  using  BOSS  than  using  either  Legacy  or  FISH.  Additionally,  a  cueing  effect 
was  found  for  workload.  Pilots  reported  that  workload  was  significantly  lower  when  symbology 
was  paired  with  the  TSAS  and  Aural  cueing  display  than  when  paired  with  the  Aural  cueing 
display  (table  7). 
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Table  6. 

Summary  of  subjective  data  comparing  symbology  sets. 


Approach  to  landing 

Mean 

/;-value  (p  <  .05) 

Measure 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Cooper  Harper 

4.69 

3.44 

4.25 

0.04 

0.48 

0.08 

Bedford 

5.25 

3.94 

4.78 

0.04 

0.40 

0.05 

VCI  Attitude 

3.80 

2.53 

2.77 

0.03 

0.12 

0.40 

VCI  Horizontal 

3.44 

2.00 

2.69 

0.017 

0.12 

0.21 

VCI  Vertical 

3.59 

1.66 

2.50 

0.017 

0.07 

0.06 

Table  7. 

Summary  of  subjective  test  data  where  significance  was  found  between  cueing  displays. 


Approach 
to  landing 

Mean 

/;-value  (p  <  .05) 

Index 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Bedford 

4.625 

4.667 

4.833 

4.500 

0.932 

0.445 

0.581 

0.493 

0.216 

0.024 

*  “Both”  refers  to  the  combined  TSAS  and  Aural  display. 


Questionnaire  rank  data 

Table  8  shows  pilot  rank  data  sums.  The  test  pilots  ranked  the  order  of  difficulty  for  the 
approach  to  landing  maneuver  for  all  cueing  combinations  within  each  visual  symbology  set 
from  easiest  (1)  to  hardest  (4).  However,  due  to  a  change  in  the  questionnaire  fonnat  after  the 
second  test  pilot,  only  data  from  test  pilots  3  to  8  were  used  for  this  analysis.  For  the  Legacy  and 
BOSS  symbology  sets,  pilots  ranked  the  TSAS  with  Aural  combination  as  the  easiest  to  fly, 
while  for  FISH,  the  TSAS  cueing  display  was  ranked  the  easiest  to  fly.  An  overall  sum  of 
rankings  shows  that  for  the  approach  to  landing  maneuver,  the  TSAS  cueing  display  and  the 
TSAS  and  Aural  cueing  display  received  the  same  overall  score  as  the  easiest  to  fly.  The  IR 
condition  (no  supplemental  cues)  was  the  most  difficult  to  fly  overall. 

Table  8. 

Cumulative  summation  of  ranking  from  questionnaires. 


Pilot  ranking 

IR 

TSAS 

Aural 

TSAS  and  Aural 

Legacy 

21 

13 

14 

12 

BOSS 

19 

12 

18 

11 

FISH 

17 

11 

19 

13 

Total 

57 

36 

51 

36 

21 


Approach  to  landing  results  summary 

Table  9  shows  summary  results  for  the  approach  to  landing  maneuver.  Overall,  BOSS 
symbology  paired  with  Tactile  and  Aural  produced  the  best  performance  and  were  preferred  over 
other  conditions.  Subjective  data  and  rank  data  findings  agreed  that  the  test  pilots  most  preferred 
the  TSAS  and  Aural  cueing  display,  and  that  the  Aural  cueing  display  alone  was  the  least 
preferred. 


Table  9. 

Approach  to  landing  summary. 


Measure 

Symbology 

Cueing  Display 

Flight 

BOSS 

TSAS  and  Aural 

Rank 

N/A 

TSAS  and  Aural 

Subjective 

BOSS 

TSAS  and  Aural 

Overall 

BOSS 

TSAS  and  Aural 

Approach  to  hover 


Flight  performance 

In  the  approach  to  hover  maneuver,  statistical  significance  between  the  visual  symbologies 
was  found  for  the  following  measures:  position,  heading,  altitude,  and  speed  (table  10).  Cueing 
effectiveness  was  determined  using  pairwise  comparisons  for  a  symbology  set  using  the 
components  of  the  indices.  Table  1 1  shows  cueing  significance  for  BOSS  for  position,  heading, 
and  altitude.  Significance  was  found  for  FISH  symbology  for  heading  and  speed,  and  for  Legacy 
symbology  for  speed. 


Table  10. 

Mean  deviations  by  symbology  and  significance  values  between  symbologies. 


Approach  to  hover 

Mean 

/j-value  (p  <  .(J 

•5) 

Index 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Position(ft.) 

50.904 

9.681 

33.963 

<0.001 

0.006 

<0.001 

Heading  (deg.) 

6.914 

5.282 

6.626 

0.080 

1.000 

0.066 

Altitude  (ft.) 

19.965 

9.525 

10.698 

<0.001 

<0.001 

0.200 

Speed  (kts) 

7.836 

6.132 

5.364 

0.150 

0.003 

0.408 

22 


Table  11. 

Cueing  mean  deviations  and  significance  values  for  approach  to  hover. 


Approach 
to  hover 

Mean  deviation 

/7-value  (p  <  .05) 

Measure 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Position 

(ft.) 

BOSS 

(Hover) 

3.98 

1 1.796| 

6.526 

4.697 

0.012 

0.105 

0.562 

0.426 

0.137 

0.669 

Heading 

(deg.) 

BOSS 

1.13 

1.42 

0.963 

0.946 

0.224 

0.429 

0.291 

0.137 

0.031 

0.889 

FISH 

1.448 

1.329 

1.072 

0.851 

0.819 

0.141 

0.012 

0.271 

0.028 

0.289 

Altitude 

(ft.) 

BOSS 

(RMSD 

Level) 

7.363 

7.441 

8.291 

9.195 

0.921 

0.41 

0.033 

0.481 

0.128 

0.413 

BOSS 

(RMSD 

Hover) 

1.79 

1.216 

1.554 

1.249 

0.012 

0.392 

0.143 

0.63 

0.703 

0.821 

Speed 

(kts) 

FISH 

(Level) 

0.744 

1.664| 

0.756 

0.879 

0.100 

0.825 

0.337 

0.036 

0.313 

0.459 

Legacy 

(Approach) 

5.401 

6.472 

8.656 

6.971 

0.269 

0.035 

0.543 

0.303 

0.982 

0.423 

*“Both”  refers  to  the  combined  TSAS  and  Aural  display, 
f  Outlier  included  in  the  mean. 


Position 

The  index  for  approach  to  hover  position  deviation  was  derived  from  the  RMSD  of  the  lateral 
and  longitudinal  positions  during  the  hover  segment  only,  therefore  no  longitudinal  data  were 
included  in  the  analysis  for  the  level  flight,  transition  to  approach,  approach,  and  transition  to 
hover  segments.  The  lateral  position  data  during  those  same  segments  could  not  be  used  due  to 
the  differences  in  how  BOSS  and  FISH  symbologies  plot  the  course  to  the  target  hover  zone. 
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Figure  7.  Position  index  data  by  symbology  (left)  and  the  RMSD  of  the  longitudinal  position 
deviation  data  during  the  hover  segment,  for  BOSS  by  cueing  (right). 

Note:  “Both”  refers  to  the  combined  TSAS  and  Aural  display. 

Symbology.  Figure  7  (left)  shows  that  pilot  overall  position  maintenance  for  the  approach  to 
hover  maneuver  was  better  when  using  BOSS  symbology.  Statistically  significant  differences  in 
position  performance  were  found  among  all  symbology  sets.  Pilot  position  performance  was 
significantly  worse  with  Legacy  symbology  than  with  either  BOSS  or  FISH  symbologies. 

Cueing.  Figure  7  (right)  shows  a  significant  difference  between  BOSS  and  BOSS  paired  with 
TSAS.  The  results  are  inconclusive  due  to  an  outlier  existing  in  the  data. 

Heading 

The  heading  deviation  index  was  computed  from  the  SD  of  the  heading  during  the  level, 
transition  to  approach,  approach,  and  transition  to  hover  segments,  and  the  RMSD  of  the  hover 
segment. 
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Figure  8.  The  SD  of  heading  data  during  the  level  segment  for  FISFI  (left)  and  during  the 

transition  to  hover  segment  for  BOSS  (right).  Note:  “Both”  refers  to  the  combined 
TSAS  and  Aural  display. 

Symbology.  No  significant  differences  in  heading  deviation  were  found  among  symbology 
sets. 

Cueing.  Figure  8  shows  cueing  significance  for  heading.  In  the  level  phase,  significance  was 
found  for  the  FISH  symbology  and  for  the  transition  to  hover  phase  for  the  BOSS  symbology. 
During  the  level  flight  portion  of  the  maneuver,  FISH  with  the  TSAS  and  Aural  cueing  display 
was  better  than  FISH  and  FISH  paired  with  TSAS  cueing  displays. 

In  the  transition  to  hover  portion  of  the  maneuver,  performance  was  better  with  BOSS 
symbology  when  paired  with  the  TSAS  and  Aural  cueing  display  than  the  TSAS  cueing  display. 

Altitude 

The  altitude  deviation  index  was  computed  from  the  RMSD  of  the  altitude  during  the  level 
and  hover  segments  of  the  maneuver.  Note  that  BOSS  does  not  provide  altitude  command 
guidance  in  the  level  segment. 
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Figure  9.  Altitude  index  data  by  symbology  (left),  and  altitude  RMSD  for  level  and  hover 

segments  (middle  and  right).  Note:  “Both”  refers  to  the  combined  TSAS  and  Aural 
display. 

Symbology.  Figure  9  (left)  indicates  altitude  maintenance  was  better  for  BOSS  and  FISH 
symbologies  than  for  Legacy  symbology. 

Cueing.  Figure  9  (middle  and  right)  shows  cueing  significance  for  altitude.  Significant 
differences  were  found  in  both  the  level  and  hover  segments.  In  the  level  segment,  the  test  pilots 
were  better  able  to  maintain  desired  altitude  with  BOSS  than  with  BOSS  paired  with  the  TSAS 
and  Aural  cueing  display. 

Speed 

The  speed  deviation  index  was  computed  using  the  RMSD  of  the  speed  during  the  level  and 
approach  segments.  Speed  data  for  the  transition  and  hover  segments  were  not  analyzed. 
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Figure  10.  Speed  index  data  by  symbology  (left),  and  speed  RMSD  for  level  and  approach 

segments  (middle  and  right).  Note:  “Both”  refers  to  the  combined  TSAS  and  Aural 
display. 

Symbology.  Figure  10  (left)  shows  approach  to  hover  speed  deviation  data  which  indicate 
that  performance  using  FISH  symbology  was  significantly  better  than  using  Legacy  symbology. 
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Cueing.  Figure  10  (middle  and  right)  shows  speed  perfonnance  was  significantly  different  for 
both  Legacy  and  FISH  symbologies.  The  significance  between  FISH  paired  with  Aural  and 
FISH  paired  with  TSAS  occurred  in  the  level  phase  of  the  maneuver  and  might  be  attributed  to 
an  outlier  in  the  data.  The  significance  in  the  legacy  symbology  occurred  in  the  approach 
segment  of  the  maneuver.  The  data  indicate  pilots  flew  the  approach  faster  when  provided  Aural 
cueing  over  Legacy  symbology. 

Subjective  data 

Ratings  show  pilots  preferred  BOSS  symbology  over  Legacy  symbology  for  all  indices. 

Pilots  rated  BOSS  and  FISH  symbologies  over  Legacy  symbology  on  the  vertical  VCI  (table  12). 

Table  12. 

Summary  of  subjective  data. 


Approach  to  hover 

Mean 

p-value  (/;  <  .05) 

Index 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Cooper  Harper 

5.5313 

3.9063 

4.6250 

0.017 

0.324 

0.233 

Bedford 

5.5938 

4.2188 

5.1563 

0.020 

0.445 

0.128 

VCI  Attitude 

3.9478 

2.4991 

2.7706 

0.012 

0.068 

0.208 

VCI  Horizontal 

3.5625 

2.0313 

2.5938 

0.018 

0.079 

0.293 

VCI  Vertical 

3.7188 

1.7813 

2.5313 

0.018 

0.021 

0.108 

Approach  to  hover  summary 

Table  13  indicates  that  for  the  approach  to  hover  maneuver,  pilots  preferred  and  perfonned 
better  with  the  BOSS  symbology  set  over  Legacy  and  Fish.  Pilots  performed  better  with  the 
TSAS  and  Aural  cueing  display  over  other  cueing  displays,  independent  of  symbology  set. 

Table  13. 

Summary  of  symbology  sets  and  cueing  displays  for  Approach  to  Hover. 


Measure 

Symbology 

Cueing  Display 

Flight 

BOSS 

TSAS  and  Aural 

Rank 

N/A 

N/A 

Subjective 

BOSS 

N/A 

Overall 

BOSS 

TSAS  and  Aural 

Hover 


Flight  performance 

In  the  hover  maneuver,  significant  differences  were  found  between  symbology  sets  for  the 
following  measures:  position,  heading,  and  altitude  (table  14).  Table  15  shows  that,  when  using 
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the  FISH  symbology,  the  Aural  cueing  display  was  significantly  better  for  heading  control  than 
the  Aural  cueing  display  combined  with  TSAS. 

Table  14. 

Mean  deviations  by  symbology  and  significance  values  between  symbologies. 


Hover 

Mean 

p-value  (p  < .( 

•5) 

Index 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Position(ft.) 

28.447 

4.896 

14.357 

<0.001 

<0.001 

<0.001 

Heading  (deg.) 

2.028 

1.891 

1.017 

0.595 

0.024 

0.312 

Altitude  (ft.) 

3.576 

1.062 

3.019 

<0.001 

0.231 

<0.001 

Table  15. 

Cueing  mean  deviations  and  significance  values  for  Hover. 


Hover 

Mean  deviation 

i 

p-value  (n  <  .05) 

Heading 

(deg.) 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

FISH* 

0.944 

0.834 

0.753 

1.535 

0.095 

0.251 

0.661 

0.251 

0.385 

0.026* 

Both”  refers  to  the  combined  TSAS  and  Aural  display. 


Position 

For  the  hover  maneuver,  the  position  deviation  index  was  calculated  from  the  RMSD  of  the 
lateral  and  longitudinal  positions  during  the  hover. 


Hover  Position  Deviation 
Symbology 


Figure  1 1 .  Position  index  data  grouped  by  symbology. 

Symbology.  Figure  1 1  shows  significance  in  hover  position  deviation.  All  symbology  sets 
were  significantly  different  from  each  other.  BOSS  symbology  produced  the  best  performance 
of  the  three. 
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Cueing.  No  significant  difference  in  altitude  deviation  was  found  among  cueing  displays. 


Heading 


Hover  Heading  Deviation 
Symbology 


Legacy  BOSS  FISH 

Error  Bars  +/- 1  S.E. 


Hover  RMSD  Heading 
FISH  Symbology 


2.0 


IR  TSAS  Aural  Both 


Error  Bars  +/- 1  S.E. 


Figure  12.  Heading  index  data  grouped  by  symbology  (left)  and  the  RMSD  of  heading  (right). 
Note:  “Both”  refers  to  the  combined  TSAS  and  Aural  display. 

Symbology.  Figure  12  (left)  shows  significance  in  hover  heading  deviation.  Pilot  heading 
maintenance  was  statistically  better  for  the  FISH  than  the  Legacy  symbology. 

Cueing.  Figure  12  (right)  shows  cueing  significance  for  heading  during  the  hover  maneuver. 
Significance  was  found  between  FISH  paired  with  the  Aural  cueing  display  and  FISH  paired 
with  the  TSAS  and  Aural  cueing  display.  Note  that  this  outcome  is  the  result  of  an  outlier  in  the 
data  (Outlier  chart,  appendix  c). 

Altitude 

The  altitude  deviation  index  was  computed  from  the  RMSD  of  the  altitude  during  the  hover 
maneuver. 
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Hover  Altitude  Deviation 
Symbology 
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Figure  13.  Altitude  index  data  by  symbology. 

Symbology.  Figure  13  shows  altitude  maintenance  perfonnance  was  statistically  better  with 
BOSS  symbology  than  with  either  FISH  or  Legacy  symbologies. 

Cueing.  No  significant  difference  in  altitude  deviation  was  found  among  cueing  displays. 
Questionnaire  rank  data 

Pilots  ranked  the  order  of  difficulty  for  the  hover  maneuver  for  all  cueing  combinations  within 
each  visual  symbology  set  from  easiest  (1)  to  hardest  (4).  Table  16  shows  the  TSAS  cueing 
display  was  ranked  to  be  the  easiest  to  fly,  followed  by  the  TSAS  and  Aural  cueing  display.  The 
Aural  cueing  display  was  ranked  the  most  difficult  to  fly. 

Table  16. 

Summations  of  rankings  from  questionnaires. 


Pilot  Ranking 

IR 

TSAS 

Aural 

TSAS  and 
Aural 

Legacy 

20 

8 

19 

13 

BOSS 

20 

9 

19 

12 

FISH 

15 

7 

21 

17 

Total 

55 

24 

59 

32 

Subjective  data 

Pilots  preferred  the  BOSS  over  Legacy  symbology  for  all  indices.  Additionally,  the  BOSS 
symbology  was  rated  better  than  the  FISH  symbology  on  Cooper  Harper,  Bedford  Workload 
Scale,  and  the  Vertical  VCI. 
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Table  17. 

Summary  of  subjective  data  for  hover. 


Hover 

Mean 

p-value  (p  <  .05) 

Index 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Cooper  Harper 

5.1875 

3.5313 

4.562 

0.018 

0.398 

0.042 

Bedford 

5.3125 

3.7188 

5.0313 

0.020 

0.779 

0.027 

VCI  Attitude 

3.6878 

2.2497 

2.7288 

0.012 

0.161 

0.205 

VCI  Horizontal 

3.1875 

1.9375 

2.5938 

0.012 

0.293 

0.125 

VCI  Vertical 

3.500 

1.5938 

2.5625 

0.012 

0.074 

0.017 

Hover  summary 

Table  18  indicates  that  for  the  hover  maneuver,  the  BOSS  symbology  and  the  Aural  cueing 
display  and  the  TSAS  cueing  display  produced  the  best  performance  and  were  preferred  over 
other  conditions. 


Table  18. 

Summary  of  symbology  sets  and  cueing  displays  for  the  hover  maneuver. 


Measure 

Symbology 

Cueing  Display 

Flight 

BOSS 

Aural 

Rank 

N/A 

TSAS 

Subjective 

BOSS 

N/A 

Overall 

BOSS 

Aural  or  TSAS 

Sidestep 


Flight  performance 

Although  the  sidestep  maneuver  began  and  ended  with  20  s  hovers,  only  the  sidestep  segment 
data  were  analyzed. 


Table  19. 

Mean  deviations  by  symbology  and  significance  values  between  symbology  sets. 


Sidestep 

Mean 

p-value  (p  <  .05) 

Index 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Position  (ft.) 

80.340 

12.160 

13.015 

<0.001 

<0.001 

0.763 
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Table  20. 

Cueing  mean  deviations  and  significance  values  for  sidestep. 


Sidestep 

Mean  deviation 

p-value  (p  <  .05) 

Measure 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Position  (ft.) 

FISH 

10.064 

6.695 

13.122 

22.18 

0.134 

0.225 

0.019 

0.017 

0.011 

0.305 

Heading  (deg.) 

FISH 

0.838 

1.157 

0.806 

1.154 

0.258 

0.131 

0.488 

0.017 

0.635 

0.074 

Speed  (kts) 

Legacy 

5.04 

5.352 

4.988 

4.885 

0.133 

0.832 

0.15 

0.01 

0.013 

0.506 

*  “Both”  refers  to  the  combined  TSAS  and  Aural  display. 


In  the  sidestep  maneuver,  statistical  significance  between  symbologies  was  found  for  the 
position  index  (table  19).  Table  20  shows  cueing  significance  for  the  FISH  symbology  for 
position  and  heading,  and  for  Legacy  symbology  for  speed. 

Position 


For  the  sidestep  maneuver,  the  position  deviation  index  was  computed  from  the  RMSD  of  the 
lateral  position  during  the  sidestep  segment. 


Sidestep  Position  Deviation 
Symbology 
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Sidestep  RMSD  Lateral  Position 
Deviation 
FISH  Symbology 
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IR  TSAS  Aural  Both 

Error  Bars  +/- 1  S.E. 


Figure  14.  Position  index  data  grouped  by  symbology  (left)  and  the  RMSD  of  lateral  position 
deviation  data  for  FISH  grouped  by  cueing  (right).  Note:  “Both”  refers  to  the 
combined  TSAS  and  Aural  display. 

Symbology.  Figure  14  (left)  shows  pilots  performed  significantly  better  when  using  the 
BOSS  and  FISH  symbologies  than  when  using  the  Legacy  symbology. 
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Cueing.  Figure  14  (right)  shows  for  the  FISH  symbology  set,  multiple  cueing  differences 
were  found.  Performance  using  the  TSAS  cueing  display  was  significantly  better  than  the  Aural 
cueing  display  and  the  TSAS  and  Aural  cueing  display.  Additionally,  the  FISH  symbology  was 
significantly  better  than  the  TSAS  and  Aural  cueing  display. 

Heading 

The  heading  deviation  index  was  computed  from  the  heading  RMSD  during  the  sidestep 
segment  of  the  maneuver. 


Sidestep  RMSD  Heading 
Deviation 
FISH  Symbology 


1.6 


IR  TSAS  Aural  Both 

Error  Bars  +/- 1  S.E. 


Figure  15.  The  RMSD  of  heading  for  FISH  and  grouped  by  cueing.  Note:  “Both”  refers  to  the 
combined  TSAS  and  Aural  display. 

Symbology.  No  significance  was  found  in  the  visual  symbology  data. 

Cueing.  Figure  15  shows  heading  maintenance  performance  with  the  FISH  symbology  for 
sidestep  was  better  when  paired  with  the  Aural  cueing  display  than  when  paired  with  the  TSAS 
cueing  display. 

Speed 

The  speed  index  was  computed  from  the  maximum  velocity  measured  during  the  sidestep 
maneuver. 
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Sidestep  Maximum  Lateral 
Velocity 

Legacy  Symbology 

*  # 


IR  TSAS  Aural  Both 

Error  Bars  +/- 1  S.E. 


Figure  16.  The  maximum  lateral  velocity  data  for  the  Legacy  symbology  grouped  by  cueing. 
Note:  “Both”  refers  to  the  combined  TSAS  and  Aural  display. 

Symbology.  No  significance  was  found  in  the  symbology  data. 

Cueing.  The  data  indicated  in  Figure  16  shows  with  Legacy  symbology  the  sidestep  velocity 
was  faster  when  paired  with  the  TSAS  cueing  display  than  with  Aural  or  TSAS  and  Aural  cueing 
displays. 

Questionnaire  rank  data 

The  test  pilots  ranked  the  order  of  difficulty  for  the  sidestep  maneuver  for  all  cueing 
combinations  within  each  visual  symbology  set  from  easiest  (1)  to  hardest  (4).  Table  21  shows 
the  TSAS  cueing  display  was  ranked  as  the  easiest  to  fly,  followed  by  the  TSAS  and  Aural 
cueing  display.  The  Aural  cueing  display  was  ranked  the  most  difficult  to  fly. 

Table  21. 

Summations  of  rankings  from  questionnaires. 


Pilot  ranking 

IR 

TSAS 

Aural 

TSAS  and 
Aural 

Legacy 

18 

9 

19 

14 

BOSS 

18 

8 

19 

15 

FISH 

13 

7 

22 

18 

Total 

49 

24 

60 

47 

34 


Subjective  ratings 


Table  22  data  indicate  pilots  rated  the  BOSS  symbology  over  the  Legacy  symbology  for  all 
VCIs. 


Table  22. 

Summary  of  subjective  data  comparing  symbology  sets. 


Sidestep 

Mean 

p-value  (p  <  .05) 

Index 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Cooper  Harper 

4.9688 

4.1563 

4.3125 

0.079 

0.395 

0.799 

Bedford 

5.3750 

4.5000 

4.9688 

0.079 

0.528 

0.181 

VCI  Attitude 

3.7397 

2.6138 

2.6459 

0.012 

0.123 

0.499 

VCI  Horizontal 

3.0625 

2.0625 

2.3438 

0.011 

0.233 

0.671 

VCI  Vertical 

3.4688 

2.0313 

2.5313 

0.017 

0.150 

0.105 

Table  23. 

Summary  of  subjective  test  data  where  significance  was  found  between  cueing  displays. 


Sidestep 

Mean 

o-value  (p  <  .05) 

Index 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

VCI  Horizontal 

2.6667 

2.5000 

2.3333 

2.4583 

0.244 

0.027 

0.064 

0.180 

0.496 

0.461 

*  “Both”  refers  to  the  combined  TSAS  and  Aural  display. 


Symbology.  Table  22  shows  that  in  the  sidestep  maneuver,  pilots  rated  the  BOSS  symbology 
over  the  Legacy  symbology  for  all  VCIs. 

Cueing.  Table  23  shows  cueing  significance  was  found  for  the  Horizontal  VCI  rating  which 
suggests  that  pilots  preferred  the  Aural  cueing  display  over  the  IR  condition. 

Sidestep  summary 

Table  24  indicates  that  for  the  sidestep  maneuver  the  BOSS  symbology  paired  with  the  TSAS 
cueing  display  produced  the  best  performance  and  was  preferred  over  other  displays. 

Table  24. 

Summary  of  symbology  sets  and  cueing  displays. 


Measure 

Symbology 

Cueing  Display 

Flight 

BOSS 

TSAS 

Rank 

N/A 

TSAS 

Subjective 

BOSS 

Aural 

Overall 

BOSS 

TSAS 
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Biometric  data 


The  physiological  measures  of  psychological  stress  resulting  from  the  pilot’s  cognitive 
workload  were  measured  through  HRV,  GSR,  and  RR.  These  measurements  were  chosen 
because  of  “minimal  invasiveness”  and  potential  for  future  use  in  aircraft.  The  biometric  data 
were  analyzed  with  BIOP  AC’s®  AcqKnowledge  ®  and  MathWorks’®  MATLAB®  software.  Full 
biometric  results  are  located  in  appendix  d. 


Table  25. 

Summary  of  conditions  with  the  lowest  biometric  stress  responses  by  maneuver  and  measure 

based  on  mean  findings. 


Maneuver 

Mean 
Heart  Rate 

Mean  Respiratory 
Interval 

Mean  Galvanic 
Skin  Response 

Heart  Rate 
Variability  Ratio 

Approach  to  Landing 

BOSS  IR 

Legacy  TSAS 

FISH  IR 

BOSS 

TSAS  and  Aural 

Approach  to  Hover 

FISH  IR 

BOSS  IR 

FISH  TSAS 

FISH  TSAS  and 
Aural 

Hover 

FISH  IR 

Legacy  TSAS 

BOSS  Aural 

FISH  TSAS  and 
Aural 

Sidestep 

FISH  IR 

Legacy  TSAS 

BOSS  TSAS 

Legacy  TSAS 

Condition-specific  effects  for  each  significant  biometric  outcome  variable 
Mean  respiration  rate  interval 

Pairwise  post-hoc  tests  of  the  main  effect  for  condition  on  mean  respiration  interval 
(controlling  for  the  non-significant  effect  of  maneuver),  demonstrated  the  following  significant 
pairwise  differences: 

•  Legacy  TSAS  >  Legacy  IR  (Diff  =  0.11,/?  =  0.0 1 8) 

Legacy  IR  demonstrated  a  significantly  lower  mean  inter-peak-interval  for  respiration  than 
Legacy  TSAS,  which  suggests  that  pilots’  stress  levels  were  significantly  lower  in  Legacy  TSAS 
than  in  Legacy  IR  condition. 

Mean  GSR 

Pairwise  post-hoc  tests  of  the  main  effect  for  condition  on  mean  tonic  GSR  (controlling  for 
the  non-significant  effect  of  maneuver)  demonstrated  the  following  significant  pairwise 
differences: 

•  Legacy  Aural  >  BOSS  TSAS  (Diff  =  4.966,/  =  0.002) 

•  Legacy  Aural  >  BOSS  Aural  (Diff  =  4.251,/  =  0.028) 

•  Legacy  Aural  >  FISH  IR  (Diff  =  5.156,/  <  .001) 

•  Legacy  Aural  >  FISH  TSAS  (Diff  =  4.454,/  =  0.008) 
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•  Legacy  Aural  >  FISH  Aural  (Diff  =  4.675,/?  =  0.004) 

•  Legacy  Aural  >  FISH  TSAS  and  Aural  (Diff  =  4.262,/?  =  0.021) 

Legacy  Aural  demonstrated  a  Tonic  GSR  level  that  was  consistently  and  significantly  higher 
than  several  other  conditions,  indicating  that  pilots’  stress  levels  were  significantly  higher  in  this 
condition  than  BOSS  TSAS,  BOSS  Aural,  and  all  FISH  cueing  conditions.  The  differences 
between  Legacy  Aural  and  Legacy  TSAS  (3.718),  Legacy  TSAS  and  Aural,  and  BOSS  TSAS 
and  Aural  (3.460)  were  approaching  significance  {p  <  0.1). 

Heart  rate  variability 

HRY  Sympathetic:  Vagal  Proportion  Ratio.  While  not  sensitive  to  condition  effects  in  the 
present  test,  HRV  appeared  to  be  quite  sensitive  to  changes  between  maneuvers.  Like  HR,  HRV 
data  are  extracted  from  the  ECG  signal,  which  was  consistently  the  strongest  and  most  accurate 
data  collected  (it  was  very  resistant  to  movement  artifact).  HRV  is  divided  into  two  proportion 
categories  based  on  a  spectral  density  analysis  of  frequencies;  these  categories  are  Sympathetic 
Activity  (correlates  with  higher  stress  levels)  and  Vagal  Activity  (correlates  with  lower  stress 
levels).  A  ratio  of  the  proportion  of  variance  attributed  to  Sympathetic  Activity  to  the  variance 
attributed  to  Vagal  Activity  provides  a  single  measure  of  HRV;  higher  levels  of  this  ratio 
indicate  higher  levels  of  stress. 

A  post-hoc  analysis  of  the  significant  effect  for  Maneuver  on  HRV  Ratio  indicated  the 
following  significant  differences: 

•  Sidestep  <  Hover  (Diff  =  2.889,/?  <  .001) 

•  Sidestep  <  Approach  to  Hover  (Diff  =  2.346,/?  =  0.004) 

•  Sidestep  <  Approach  to  Landing  (Diff  =  3.908,/?  =  0.016) 

The  sidestep  maneuver  demonstrated  a  significantly  lower  HRV  ratio  than  all  other 
maneuvers,  which  would  suggest  that  pilots’  stress  levels  were  lowest  while  completing  this 
maneuver. 


Discussion 


It  is  important  to  note  that  these  limited  results  were  attained  from  eight  highly  experienced 
test  pilots  perfonning  flight  maneuvers  of  short  duration  during  relatively  comfortable  and  stress- 
free  simulated  flight.  As  such,  the  results  cannot  be  generalized  and  interpreted  as  applicable  to 
all  Army  aviator  experience  levels  or  to  perfonnance  during  actual  flight  conditions.  Further 
assessment  of  a  broader  array  of  aviator  experience  under  operational  conditions  (e.g.,  fatigue, 
workload)  is  recommended  to  fully  understand  how  these  displays  and  cueing  combinations 
might  perform  under  real-world  conditions.  Final  determinations  of  the  operational  significance 
of  these  findings  (i.e.,  vs.  statistical  significance)  are  deferred  to  the  test  sponsor. 
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Approach  to  landing 


The  results  for  the  approach  to  landing  maneuver  indicate  that  for  position  maintenance, 

BOSS  symbology  perfonned  significantly  better  than  Legacy  symbology  by  an  average  of 
approximately  12  ft.  In  other  words,  pilots  flying  with  Legacy  symbology  landed  an  average  of 
12  ft.  further  from  the  intended  touchdown  point  than  pilots  flying  the  BOSS  symbology — an 
operationally  significant  difference.  In  addition,  greater  performance  was  observed  when  pilots 
utilized  BOSS  symbology  when  paired  with  either  the  TSAS  or  Aural  cueing  displays. 

Heading  maintenance  perfonnance  was  significantly  better  with  BOSS  symbology  when 
compared  with  Legacy  and  FISH,  although  less  than  1°  of  difference.  It  is  suspected  that  this  is 
due  to  the  presence  of  a  color  heading  tape  within  the  BOSS  symbology  which  is  easily 
referenced  by  the  pilot.  In  the  Legacy  condition,  heading  maintenance  was  better  with  the  TSAS 
and  Aural  cueing  display  than  with  the  Aural  cueing  display,  indicating  a  beneficial  synergy 
between  tactile  and  aural  cueing. 

Altitude  and  speed  maintenance  was  better  with  FISH  (approximately  2  ft.)  and  BOSS 
(approximately  3  knots)  symbologies,  respectively,  than  with  Legacy  symbology.  These  results 
likely  highlight  the  benefits  of  command  guidance  in  advanced  flight  symbologies. 

Level  segment  speed  when  using  Legacy  symbology  was  better  alone  than  with  the  Aural 
cueing  display.  This  reflects  test  pilot  comments  that  the  aural  cues  were  distracting  and 
annoying. 

When  using  the  Legacy  symbology,  pilots  had  a  slower  (i.e.,  better)  touchdown  speed  with 
TSAS  cueing  than  when  the  Legacy  symbology  was  paired  with  the  Aural  cueing  display.  This 
is  likely  attributed  to  the  cueing  synergy  of  tactile  and  visual  cues  which  provided  a  multisensory 
awareness  (a  feeling)  of  rate  of  closure  when  using  the  Legacy  symbology,  which  lacked  the 
speed  guidance  provided  by  BOSS  and  FISH. 

The  test  pilots  ranked  the  order  of  difficulty  for  approach  to  landing  for  each  cueing  display 
within  each  symbology  set.  Overall,  the  TSAS  and  the  TSAS  and  Aural  cueing  displays  were 
ranked  to  be  the  easiest  to  fly  the  approach  to  landing  (Figure  17).  These  results  support  the 
impression  that  supplemental  cueing  enhances  pilot  confidence  and  situational  awareness. 
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Approach  to  Landing  Rank  Data 


IR  TSAS  Aural  Both 

Cueing  Condition 


Figure  17.  Sum  of  pilots’  rank  scores  for  approach  to  landing.  Note:  “Both”  refers  to  the 
combined  TSAS  and  Aural  display. 

Biometric  data  showed  no  consistent  relationship  to  the  approach  to  landing  measures.  While 
mean  HR  reflected  the  favorable  performance  of  BOSS,  HRV  data  supported  test  pilot  comments 
favoring  BOSS  with  TSAS  and  Aural  cueing. 

For  all  approach  to  landing  subjective  measures,  pilots  rated  BOSS  better  than  Legacy.  The 
Bedford  Workload  Scale  showed  that  pilots’  perceived  workload  was  less  when  using  BOSS 
than  Legacy  or  FISH  symbologies.  The  data  indicated  overall  BOSS  symbology  and  the  TSAS 
and  Aural  cueing  display  produced  the  best  performance  and  were  preferred.  The  test  pilots 
reported  that  performance  using  BOSS  was  enhanced  by  supplemental  cues  and  reported  having 
sufficient  spare  capacity  to  perform  other  tasks. 

Approach  to  hover 

The  results  for  the  approach  to  hover  maneuver  indicate  that  for  position  maintenance,  BOSS 
performed,  on  average,  significantly  better  (9.7  ft.  from  the  desired  hover  position)  than  both  the 
Legacy  and  FISH  symbologies  (50.9  ft.  and  33.9  ft.,  respectively).  This  substantial  difference  is 
likely  due  to  the  intuitive  transition  design  features  (Approach  to  Hover)  of  the  BOSS 
symbology. 

Lateral  position  with  BOSS  symbology  appeared  better  when  paired  only  with  the  IR  sensor 
alone.  However,  the  results  are  inconclusive  due  to  an  outlier  that  existed  in  the  data.  Because  of 
the  small  number  of  participants  (n  =  8),  the  decision  was  made  to  not  remove  outliers  from  the 
data  when  they  were  analyzed.  To  interpret  the  findings,  it  was  necessary  to  explore  individual 
pilot  performance.  In  this  exploration,  it  was  found  that  one  pilot’s  RMSD  data  for  longitudinal 
position  was  more  than  three  standard  deviations  from  the  mean,  and  the  pilot  performed  the 
approach  out  of  standards  and  did  not  correct  hover  position.  All  other  pilots’  position 
performances  in  the  TSAS  condition  were  comparable  to  the  IR  condition.  The  outlier  chart  is  in 
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appendix  A.  Additional  research  is  needed  with  a  larger  number  of  participants  to  determine  if 
significance  actually  exists  between  the  cueing  conditions. 

Heading  data  for  BOSS  and  FISH  during  the  transition  to  hover  and  level  flight  segment 
indicate  that  heading  maintenance  was  better  for  the  TSAS  and  Aural  cueing  display  (although 
within  just  1°)  over  the  TSAS  cueing  display.  It  is  suspected  that  TSAS  and  Aural  cues 
enhanced  awareness  of  heading  maintenance  contributing  to  better  performance. 

Results  for  altitude  maintenance  during  the  approach  to  hover  maneuver  indicate  altitude 
performance  was  better  using  either  BOSS  or  FISH  than  it  was  with  Legacy.  Hover  segment 
altitude  maintenance  with  BOSS  symbology  was  better  when  paired  with  the  TSAS  cueing 
display,  over  BOSS  paired  with  only  an  IR  sensor.  Notably,  for  the  level  segment,  BOSS  altitude 
maintenance  was  better  when  paired  with  only  an  IR  sensor  compared  with  the  TSAS  and  Aural 
cueing  display.  The  results  may  suggest  that  the  parameters  for  altitude  maintenance  cueing  were 
set  too  close  causing  pilot  distraction  from  excessive  aural  cueing. 

Results  for  speed  performance  for  the  approach  to  hover  maneuver  indicate  that  performance 
with  FISH  was  significantly  better  than  Legacy  by  approximately  2.5  knots.  Speed  data  during 
the  level  segment  when  using  FISH  indicate  that  speed  maintenance  was  better  with  the  aural 
cueing  display  than  with  the  TSAS  cueing  display  due  to  “check  speed”  aural  cueing.  TSAS 
does  not  provide  speed  cueing  during  the  level  segment. 

The  test  pilots  ranked  BOSS  over  Legacy  for  all  indices.  Additionally,  the  test  pilots  ranked 
FISH  over  Legacy  for  the  Vertical  VCI.  As  with  the  Approach  to  Landing,  these  results  likely 
highlight  the  benefits  of  command  guidance  provided  in  advanced  flight  symbologies. 

Biometric  data  failed  to  consistently  predict  flight  perfonnance  or  test  pilot  ratings.  Mean 
respiratory  interval  indicated  lower  stress  during  the  BOSS  IR  condition,  but  various 
combinations  involving  FISH  showed  lower  stress  levels  that  correlated  only  with  speed 
performance. 

To  summarize,  overall  approach  to  hover  data  indicate  the  preferred  visual  symbology  was 
BOSS  and  the  preferred  cueing  display  was  TSAS  with  Aural ;  these  produced  the  best 
performance  and  were  preferred  over  other  symbologies  and  cueing  conditions. 

Hover 

On  average,  the  data  indicate  hover  position  performance  was  best  with  BOSS  symbology 
(4.9  ft.  from  desired)  over  both  Legacy  (28.5  ft.)  and  FISH  (14.4  ft.).  This  is  likely  to  due  to 
simplicity  and  intuitive  attributes  of  BOSS  symbology,  specifically,  the  home  plate  symbol. 

Heading  maintenance  data  indicate  perfonnance  with  FISH  was  significantly  better  than 
Legacy,  albeit  by  merely  1°.  Supplemental  aural  and  tactile  cues  significantly  improved 
performance  when  utilizing  FISH  symbology. 
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Hover  altitude  performance  data  indicate  BOSS  was  significantly  better  (by  approximately  2 
ft.)  than  both  FISH  and  Legacy  symbologies,  probably  due  to  the  BOSS  hover  cue  reference 
turning  green  when  the  desired  altitude  is  achieved.  This  color  change  represents  an  additional 
perceptual  cue  to  the  pilot. 

Biometric  data  failed  to  consistently  correlate  with  flight  performance  or  test  pilot  ratings  for 
the  Hover  maneuver.  Only  GSR  indicated  lower  workload/stress  with  BOSS  symbology. 

Overall,  in  the  hover  maneuver,  the  TSAS  condition  was  ranked  to  be  the  easiest  to  fly  (figure 
18).  The  BOSS  symbology  was  rated  better  than  Legacy  on  all  subjective  scales.  Additionally, 
the  BOSS  symbology  was  rated  better  than  FISH  on  Cooper  Harper,  Bedford  Workload  Scale 
Workload,  and  the  Vertical  VCI. 


Hover  Rank  Data 


IR  TSAS  Aural  Both 

Cueing  Condition 


Figure  18.  Sum  of  pilots’  rank  scores  for  hover.  Note:  “Both”  refers  to  the  combined  TSAS  and 
Aural  display. 


Sidestep 

For  the  sidestep  maneuver,  the  position  deviation  index  data  indicate  performance  was 
significantly  worse  with  Legacy  symbology  (80  ft.  from  desired  track)  than  both  BOSS  (12  ft.) 
and  FISH  (13  ft.)  symbologies.  This  was  believed  to  be  principally  due  to  the  advanced  features 
of  the  BOSS  symbology  (displaying  a  scaled  velocity  vector)  and  FISH  symbology  (via  a 
slewable  reposition  capability).  The  data  indicate  FISH  position  maintenance  was  better  with 
TSAS  (likely  due  to  the  tactile  cueing  confirming  the  direction  of  the  lateral  sidestep  maneuver) 
than  with  the  Aural  and  TSAS  and  Aural  cueing  displays.  Aural  cueing  during  the  sidestep 
maneuver  appeared  to  be  unnecessary  and  posed  a  distraction  in  the  manner  used  in  this  test. 

The  data  indicate  performance  with  FISH  symbology  was  better  when  paired  with  Aural  than 
with  TSAS.  The  speed  data  indicate  when  using  Legacy  symbology  the  sidestep  velocity  was 
faster  when  paired  with  TSAS  cueing  display  rather  than  with  the  Aural  or  the  TSAS  and  Aural 
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cueing  displays.  This  is  likely  due  to  high  degree  of  pilot  confidence  associated  with  sensing 
tactile  directional  cues  in  the  desired  track. 

Figure  19  shows  the  pilots’  rank  order  of  difficulty  for  the  sidestep  maneuver  for  each  cueing 
display  within  each  symbology  set.  Overall,  the  test  pilots  ranked  the  TSAS  cueing  display  as 
the  easiest  to  fly  and  the  Aural  cueing  display  as  the  most  difficult. 


Sidestep  Rank  Data 


IR  TSAS  Aural  Both 

Cueing  Condition 


Figure  19.  Sum  of  pilots’  rank  scores  for  sidestep.  Note:  “Both”  refers  to  the  combined  TSAS 
and  Aural  display. 

Biometric  data  failed  to  consistently  predict  perfonnance  or  test  pilot  ratings,  except  that  three 
of  the  four  biometric  measures  correctly  reflected  test  pilot  preference  for  TSAS  in  the  sidestep 
maneuver. 

The  data  indicate  that  pilots  rated  BOSS  over  Legacy  for  all  Visual  Cue  Indices.  Additional 
cueing  significance  was  found  for  the  Horizontal  VCI  rating  that  suggests  pilots  preferred  the 
Aural  cueing  display  over  the  IR  sensor  image  alone.  Overall,  in  the  sidestep  maneuver  the  data 
suggest  BOSS  symbology  and  the  TSAS  cueing  display  were  most  effective. 


Conclusions 


To  review,  the  goals  of  this  project  were  to: 

1)  Determine  if  combining  selected  symbology/cueing  sets  would  improve  flight 
performance  and/or  reduce  workload/stress. 

2)  Evaluate  if  the  effectiveness  of  different  combinations  of  the  symbology/cueing  sets 
would  be  reflected  in  test  pilot  subjective  evaluations,  flight  performance,  and  workload/stress. 

3)  Determine  if  the  effectiveness  of  different  combinations  of  symbology/cueing  sets 
would  vary  with  the  flight  task. 
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In  general,  the  results  indicate  that  test  pilot  flight  perfonnance  was  improved  when  using 
advanced  visual  symbologies,  particularly  when  combined  with  a  supplemental  form  of  cueing 
(aural  and/or  tactile).  Advanced  visual  symbologies  outperformed  Legacy  symbology  for  almost 
all  maneuvers.  BOSS  symbology  provided  better  overall  performance  compared  to  the  FISH 
symbology,  and  BOSS  symbology  was  the  most  preferred  based  on  the  subjective  ratings  for  all 
maneuvers.  These  highly  experienced  test  pilots  reported  that  the  BOSS  symbology  was  simple 
to  fly  and  provided  them  with  spare  cognitive  capacity.  Its  features  included  an  easily 
interpreted,  positive  affirmation  of  pilot  control  inputs  by  providing  a  color  change  (green  fill)  on 
the  vertical  and  horizontal  velocity  indicators  to  indicate  maneuver  accuracy.  The  test  pilots 
reported  this  function  provided  increased  awareness  of  perfonnance  and  assisted  with  recoveries 
from  flight  deviations. 

The  optimal  cueing  display  was  dependent  on  the  maneuver  and  the  type  of  visual  symbology 
being  used.  The  results  indicated  that  the  combined  TSAS  and  Aural  cueing  display  provided 
the  best  synergy  for  the  approach  to  landing  and  approach  to  hover  maneuvers,  while  the  TSAS 
cueing  display  was  the  optimal  supplemental  cue  for  the  hover  and  sidestep  maneuvers.  Rank 
data  indicated  that  the  test  pilots  expressed  an  overall  preference  for  the  TSAS  cueing  display, 
followed  by  the  TSAS  and  Aural  cueing  display.  The  Aural  cueing  display  (without  TSAS),  as 
implemented  in  this  study,  was  ranked  as  the  most  difficult  to  fly.  The  test  pilots  reported  that 
they  preferred  the  aural  cues  that  provided  situational  information,  such  as  the  “assume 
guidance”  and  altitude  countdown  cues,  over  aural  cues  that  demanded  corrective  action  to 
satisfy  a  required  performance  measure.  Their  opinion  was  that  excessive  aural  cues, 
particularly  those  that  demanded  corrective  action,  were  annoying  and  had  the  potential  to 
distract  and  possibly  degrade  flight  performance  (comments  supported  by  the  biometric 
monitoring  data).  When  aural  cues  were  paired  with  the  FISH  and  Legacy  symbologies,  test 
observers  witnessed  an  increase  in  test  pilot  induced  oscillations.  According  to  the  test  pilots, 
the  pilot-induced  oscillations  occurring  when  using  the  FISH  symbology  and  aural  cues  were  the 
result  of  the  visual  symbology  being  “overly”  sensitive  and  the  aural  cues  providing  insufficient 
information. 

The  results  of  the  biometric  data  failed  to  consistently  correlate  with  flight  performance  or  test 
pilot  workload  ratings  or  cueing  preferences.  Mean  respiratory  interval  data  revealed  that  for  the 
Legacy  symbology,  TSAS  was  preferred  over  Legacy  without  supplemental  cueing.  GSR  results 
indicate  that  the  Legacy  symbology  with  aural  cueing  was  indicative  of  increased  stress  over 
BOSS  with  TSAS,  BOSS  with  Aural,  and  all  FISH  conditions.  The  biometric  measures  may 
reflect  aspects  of  workload  and  perfonnance  that  are  not  fully  understood.  At  this  point, 
conclusions  based  on  these  biometric  parameters  would  be  premature. 


Recommendations 


If  a  project  management  decision  is  anticipated  based  on  these  results,  it  is  recommended  that 
future  tests  in  this  program  include  BOSS  with  3D  conformal  symbology  and  sensor  imagery.  If 
proprietary  restrictions  permit,  BOSS  symbology  should  incorporate  both  the  FISH  enroute  page 
attributes,  and  the  FISH  sidestep  slewing  capabilities.  BOSS  symbology  should  be  paired  with 
TSAS  and  modified  aural  cues  for  all  maneuvers.  TSAS  modes  should  include  a  precision 
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approach  mode  and  threat  cues.  Aural  cues  should  provide  altitude  infonnation  and  indicate 
mode  changes. 

Further  exploration  of  biometric  stress  measures  is  suggested  to  provide  baseline  data  for 
future  flight  tests  and  to  determine  which  measures  provide  the  most  robust  data  in  dynamic 
flight  environments.  To  reduce  the  noise  in  the  GSR  data  caused  by  hand  movements,  it  is 
recommended  that  alternate  electrode  placements  be  explored. 
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Appendix  A. 

Flight  performance  and  subjective  results. 


Table  A-l. 

Mean  deviations  by  symbology  and  significance  values  between  symbology  sets  for  Approach  to 

Landing  maneuver. 


Approach  to  Landing 

Mean 

/7-value  (p  <  .05) 

Index 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Position(feet) 

22.683 

9.823 

15.465 

<0.001 

0.075 

0.151 

Heading  (degrees) 

3.656 

2.661 

3.900 

0.001 

0.960 

<0.001 

Altitude  (feet) 

9.849 

8.816 

7.967 

0.270 

0.007 

0.381 

Speed  (knots) 

9.946 

6.703 

8.184 

0.003 

0.271 

0.225 

Table  A-2. 

Cueing  mean  deviations  and  significance  values  for  Approach  to  Landing  maneuver. 


Approach  to  Landing 

Mean  Deviation 

o-value  (p  <  .C 

15) 

Measure 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Position  (feet) 

Legacy 

(Distance  at  Touchdown) 

17.231 

20.019 

29.064 

24.419 

0.493 

0.138 

0.675 

0.333 

0.947 

0.254 

BOSS 

(Distance  at  Touchdown) 

10.270 

8.163 

6.941 

13.920 

0.030 

0.048 

0.347 

0.734 

0.439 

0.443 

FISH 

(Distance  at  Touchdown) 

24.632 

13.476 

6.961 

16.789 

0.612 

0.135 

0.467 

0.085 

0.968 

0.276 

Heading  (degrees) 

Legacy 

(Level  Phase  SD) 

0.750 

0.534 

0.888 

0.353 

0.229 

0.672 

0.144 

0.183 

0.949 

0.045 

BOSS 

(Level  Phase  SD) 

0.253 

0.415 

0.361 

0.282 

0.362 

0.667 

0.970 

0.659 

0.362 

0.531 

FISH 

(Level  Phase  SD) 

0.848 

0.886 

1.214 

1.062 

0.953 

0.135 

0.601 

0.179 

0.704 

0.332 

Legacy 

(Transition  to  Approach  SD) 

0.828 

0.446 

0.803 

0.755 

0.413 

0.978 

0.888 

0.472 

0.366 

0.891 

BOSS 

(Transition  to  Approach  SD) 

0.417 

0.464 

0.441 

0.280 

0.665 

0.944 

0.304 

0.618 

0.157 

0.151 

FISH 

(Transition  to  Approach  SD) 

1.154 

0.612 

1.352 

1.183 

0.189 

0.828 

0.628 

0.117 

0.423 

0.408 

Legacy 

(Approach  SD) 

1.060 

0.833 

0.762 

0.956 

0.443 

0.254 

0.822 

0.199 

0.373 

0.164 
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Table  A-2  (continued). 


Approach  to  Landing 

Mean  D 

•eviation 

p-value  (/;  <  X 

>5) 

Measure 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Heading  (degrees) 

BOSS 

(Approach  SD) 

1.000 

0.813 

0.732 

0.935 

0.480 

0.278 

0.737 

0.341 

0.409 

0.216 

FISH 

(Approach  SD) 

1.164 

1.267 

1.239 

1.382 

0.675 

0.964 

0.235 

0.598 

0.486 

0.343 

Legacy 

(Transition  to  Touchdown  SD) 

1.300 

1.696 

1.261 

1.401 

0.197 

0.813 

0.975 

0.340 

0.497 

0.768 

BOSS 

(Transition  to  Touchdown  SD) 

0.782 

1.335 

0.870 

1.261 

0.121 

0.431 

0.332 

0.162 

0.202 

0.523 

FISH 

(Transition  to  Touchdown  SD) 

0.489 

0.547 

0.614 

0.591 

0.852 

0.835 

0.251 

0.711 

0.274 

0.642 

Altitude  (feet) 

Legacy 

(Level  Phase  RMSD) 

9.468 

10.609 

10.150 

9.168 

0.324 

0.541 

0.876 

0.752 

0.157 

0.518 

BOSS 

(Level  Phase  RMSD) 

8.860 

9.218 

8.681 

8.503 

0.545 

0.888 

0.851 

0.490 

0.330 

0.926 

FISH 

(Level  Phase  RMSD) 

8.001 

7.017 

7.756 

9.093 

0.226 

0.659 

0.447 

0.438 

0.100 

0.314 

Speed  (knots) 

Legacy 

(Level  Phase  RMSD) 

0.391 

0.586 

1.361 

0.829 

0.522 

0.017 

0.404 

0.096 

0.607 

0.054 

BOSS 

(Level  Phase  RMSD) 

1.085 

1.691 

1.070 

0.728 

0.389 

0.416 

0.127 

0.065 

0.119 

0.745 

FISH 

(Level  Phase  RMSD) 

0.919 

1.140 

1.112 

1.632 

0.844 

0.724 

0.129 

0.860 

0.061 

0.278 

Legacy 

(Approach  RMSD) 

8.510 

6.521 

7.369 

6.888 

0.322 

0.449 

0.389 

0.616 

0.738 

0.763 

BOSS 

(Approach  RMSD) 

5.149 

5.366 

4.916 

5.640 

0.665 

0.982 

0.569 

0.593 

0.833 

0.464 

FISH 

(Approach  RMSD) 

5.302 

4.190 

4.713 

4.442 

0.398 

0.893 

0.708 

0.524 

0.777 

0.736 

Legacy 

(Landing) 

3.234 

2.655 

3.863 

2.551 

0.279 

0.538 

0.215 

0.048 

0.961 

0.237 

BOSS 

(Landing) 

1.541 

1.532 

1.173 

2.052 

0.580 

0.472 

0.227 

0.970 

0.061 

0.075 

FISH 

(Landing) 

3.759 

3.209 

2.507 

4.684 

0.345 

0.124 

0.841 

0.543 

0.277 

0.101 
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Table  A-2  (continued). 


Approach  to  Landing 

Mean  Deviation 

p-value  (p  <  .C 

»5) 

Measure 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Speed  (knots) 

Legacy 

(Landing  Lateral  Drift) 

0.322 

0.414 

0.161 

0.127 

0.375 

0.070 

0.046 

0.048 

0.060 

0.305 

BOSS 

(Landing  Lateral  Drift) 

0.279 

0.170 

0.342 

0.078 

0.648 

0.257 

0.382 

0.130 

0.092 

0.815 

FISH 

(Landing  Lateral  Drift) 

0.325 

0.308 

0.212 

0.282 

0.949 

0.342 

0.986 

0.259 

0.938 

0.457 

*  “Both”  refers  to  the  combined  TSAS  and  Aural  display. 


Table  A-3. 

Summary  of  subjective  data  for  symbology  for  Approach  to  Landing  maneuver. 


Approach  to  Landing 

Mean 

p-value  (p  <  .05) 

Measure 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Cooper  Harper 

4.6988 

3.438 

4.250 

0.042 

0.484 

0.078 

Bedford 

5.250 

3.938 

4.781 

0.035 

0.401 

0.046 

VCI  Attitude 

3.802 

2.530 

2.771 

0.028 

0.123 

0.398 

VCI  Horizontal 

3.438 

2.000 

2.688 

0.017 

0.123 

0.207 

VCI  Vertical 

3.594 

1.656 

2.500 

0.017 

0.068 

0.061 

Table  A-4. 

Summary  of  subjective  data  for  cueing  for  Approach  to  Landing  maneuver. 


Approach  to 
Landing 

Mean 

/;-value  (p  <  .05) 

Measure 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Cooper  Harper 

4.167 

3.958 

4.167 

4.208 

0.279 

0.739 

0.916 

0.157 

0.168 

0.785 

Bedford 

4.625 

4.667 

4.833 

4.500 

0.932 

0.445 

0.581 

0.493 

0.216 

0.024 

VCI  Attitude 

2.985 

3.014 

3.041 

3.097 

0.916 

0.715 

0.174 

1.000 

0.500 

0.344 

VCI  Horizontal 

2.792 

2.542 

2.667 

2.833 

0.344 

0.414 

0.916 

0.593 

0.670 

0.458 

VCI  Vertical 

2.625 

2.625 

2.500 

2.583 

0.932 

0.680 

0.730 

0.673 

0.655 

0.786 

*  “Both”  refers  to  the  combined  TSAS  and  Aural  display. 


52 


Table  A-5. 

Mean  deviations  by  symbology  and  significance  values  between  symbology  sets  for  Approach  to 

Hover  maneuver. 


Approach  to  Hover 

Mean 

p-value  (p  <  .C 

>5) 

Index 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Position(feet) 

50.904 

9.681 

33.963 

<0.001 

0.006 

<0.001 

Heading  (degrees) 

6.914 

5.282 

6.626 

0.080 

1.000 

0.066 

Altitude  (feet) 

19.965 

9.525 

10.698 

<0.001 

<0.001 

0.200 

Speed  (knots) 

7.836 

6.132 

5.364 

0.150 

0.003 

0.408 

Table  A-6. 

Cueing  mean  deviations  and  significance  values  for  Approach  to  Hover  maneuver. 


Approach  to  Hover 

Mean  Deviation 

p-value  (p  <  .1 

15) 

Measure 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Position  (feet) 

Legacy 

(Lateral  Hover  Position 
RMSD) 

7.296 

9.346 

9.865 

7.568 

0.780 

0.915 

0.610 

0.597 

0.994 

0.687 

BOSS 

(Lateral  Hover  Position 
RMSD) 

2.214 

2.377 

2.427 

4.706 

0.832 

0.538 

0.057 

0.837 

0.087 

0.205 

FISH 

(Lateral  Hover  Position 
RMSD) 

1.990 

2.142 

2.144 

2.832 

0.845 

0.985 

0.347 

0.834 

0.197 

0.211 

Legacy 

(Longitudinal  Hover 
Position  RMSD) 

42.870 

33.194 

34.107 

59.372 

0.877 

0.618 

0.471 

0.502 

0.521 

0.295 

BOSS’ 

(Longitudinal  Hover 
Position  RMSD) 

3.980 

11.796* 

6.526 

4.697 

0.012 

0.105 

0.562 

0.426 

0.137 

0.669 

FISH 

(Longitudinal  Hover 
Position  RMSD) 

34.169 

31.710 

42.445 

18.311 

0.093 

0.401 

0.248 

0.788 

0.785 

0.663 

Heading  (degrees) 

Legacy 

(Level  Phase  SD) 

0.561 

0.669 

0.693 

0.670 

0.946 

0.274 

0.649 

0.444 

0.711 

0.439 

BOSS 

(Level  Phase  SD) 

0.604 

0.317 

0.328 

0.285 

0.393 

0.126 

0.281 

0.485 

0.582 

0.770 

FISH 

(Level  Phase  SD) 

1.448 

1.329 

1.072 

0.851 

0.819 

0.141 

0.012 

0.271 

0.028 

0.289 
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Table  A-6  (continued). 


Approach  to  Hover 

Mean  Deviation 

p-value  (p  <  .C 

>5) 

Measure 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Heading  (degrees) 

Legacy  (Transition  to 
Approach  SD) 

0.862 

0.842 

0.563 

0.895 

0.416 

0.103 

0.307 

0.544 

0.800 

0.903 

BOSS  (Transition  to 
Approach  SD) 

0.815 

0.425 

0.467 

0.478 

0.299 

0.318 

0.125 

0.624 

0.725 

0.976 

FISH  (Transition  to 
Approach  SD) 

0.932 

1.267 

0.979 

0.700 

0.694 

0.409 

0.230 

0.319 

0.180 

0.782 

Legacy 

(Approach  SD) 

0.962 

0.888 

0.805 

0.848 

0.662 

0.646 

0.799 

0.923 

0.713 

0.763 

BOSS 

(Approach  SD) 

0.834 

1.055 

0.798 

0.894 

0.363 

0.762 

0.974 

0.062 

0.162 

0.529 

FISH 

(Approach  SD) 

0.870 

1.171 

1.047 

1.146 

0.209 

0.463 

0.315 

0.929 

0.952 

0.966 

Legacy 

(Transition  to  Hover  SD) 

1.645 

1.588 

1.488 

1.673 

0.638 

0.495 

0.786 

0.991 

0.792 

0.749 

BOSS 

(Transition  to  Hover  SD) 

1.130 

1.420 

0.963 

0.946 

0.224 

0.429 

0.291 

0.137 

0.031 

0.889 

FISH 

(Transition  to  Hover  SD) 

1.004 

0.744 

0.939 

1.054 

0.166 

0.782 

0.668 

0.090 

0.060 

0.456 

Legacy 

(Hover  RMSD) 

2.579 

3.426 

3.383 

2.617 

0.233 

0.176 

0.894 

0.867 

0.204 

0.167 

BOSS 

(Hover  RMSD) 

2.054 

2.701 

1.981 

2.635 

0.321 

0.423 

0.896 

0.074 

0.409 

0.465 

FISH 

(Hover  RMSD) 

2.895 

2.406 

2.190 

2.459 

0.718 

0.388 

0.366 

0.617 

0.737 

0.831 

Altitude  (feet) 

Legacy 

(Level  Phase  RMSD) 

9.745 

9.004 

11.343 

9.235 

0.842 

0.519 

0.937 

0.259 

0.854 

0.294 

BOSS 

(Level  Phase  RMSD) 

7.363 

7.441 

8.291 

9.195 

0.921 

0.410 

0.033 

0.481 

0.128 

0.413 

FISH 

(Level  Phase  RMSD) 

7.003 

7.672 

7.060 

6.734 

0.523 

0.990 

0.955 

0.568 

0.446 

0.951 

Legacy 

(Hover  RMSD) 

10.340 

8.541 

10.516 

11.136 

0.217 

0.772 

0.777 

0.373 

0.584 

0.981 

BOSS 

(Hover  RMSD) 

1.790 

1.216 

1.554 

1.249 

0.012 

0.392 

0.143 

0.630 

0.703 

0.821 

FISH 

(Hover  RMSD) 

3.693 

3.797 

3.839 

2.992 

0.761 

0.805 

0.320 

0.664 

0.202 

0.518 
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Table  A-6  (continued). 


Approach  to  Hover 

Mean  Deviation 

p-value  (p  <  .C 

>5) 

Measure 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Speed  (knots) 

Legacy 

(Level  Phase  RMSD) 

0.886 

0.753 

1.316 

0.889 

0.546 

0.311 

0.942 

0.060 

0.342 

0.121 

BOSS 

(Level  Phase  RMSD) 

0.998 

1.249 

1.008 

0.946 

0.319 

0.920 

0.652 

0.321 

0.249 

0.760 

FISH* 

(Level  Phase  RMSD) 

0.744 

1.664* 

0.756 

0.879 

0.100 

0.825 

0.337 

0.036 

0.313 

0.459 

Legacy 

(Approach  RMSD) 

5.401 

6.472 

8.656 

6.971 

0.269 

0.035 

0.543 

0.303 

0.982 

0.423 

BOSS 

(Approach  RMSD) 

5.061 

5.522 

4.567 

5.175 

0.815 

0.636 

0.551 

0.362 

0.933 

0.267 

FISH 

(Approach  RMSD) 

4.318 

4.483 

4.302 

4.310 

0.327 

0.470 

0.834 

0.214 

0.605 

0.839 

*  “Both”  refers  to  the  combined  TSAS  and  Aural  display. 


Table  A-7. 

Summary  of  subjective  data  for  symbology  for  Approach  to  Hover  maneuver. 


Approach  to  Hover 

Mean 

p-value  (p  <  .05) 

Measure 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Cooper  Harper 

5.5313 

3.9063 

4.6250 

0.017 

0.324 

0.233 

Bedford 

5.5938 

4.2188 

5.1563 

0.020 

0.445 

0.128 

VCI  Attitude 

3.9478 

2.4991 

2.7706 

0.012 

0.068 

0.208 

VCI  Horizontal 

3.5625 

2.0313 

2.5938 

0.018 

0.079 

0.293 

VCI  Vertical 

3.7188 

1.7813 

2.5313 

0.018 

0.021 

0.108 

Table  A-8. 

Summary  of  subjective  data  for  cueing  Approach  to  Hover  maneuver. 


Approach  to  Hover 

Mean 

i 

p-value  (p  <  .05) 

Measure 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Cooper  Harper 

4.542 

4.708 

4.750 

4.750 

0.233 

0.246 

0.395 

1.000 

0.861 

0.863 

Bedford 

4.875 

5.000 

5.125 

4.958 

0.680 

0.306 

0.524 

0.621 

0.832 

0.416 

VCI  Attitude 

3.166 

3.124 

3.028 

2.972 

0.674 

0.345 

0.075 

0.462 

0.248 

0.893 

VCI  Horizontal 

2.750 

2.708 

2.792 

2.667 

1.000 

0.705 

0.453 

0.581 

0.609 

0.340 

VCI  Vertical 

2.708 

2.750 

2.583 

2.667 

0.865 

0.343 

0.734 

0.496 

0.414 

0.719 

Both”  refers  to  the  combined  TSAS  and  Aural  display. 
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Table  A-9. 

Mean  deviations  by  symbology  and  significance  values  between  symbology  sets  for  Hover 

maneuver. 


Hover 

Mean 

p-value  (p< . C 

15) 

Index 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Position(feet) 

28.447 

4.896 

14.357 

<0.001 

<0.001 

<0.001 

Heading  (degrees) 

2.028 

1.891 

1.017 

0.595 

0.024 

0.312 

Altitude  (feet) 

3.576 

1.062 

3.019 

<0.001 

0.231 

<0.001 
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Table  A- 10. 

Cueing  mean  deviations  and  significance  values  for  Hover  maneuver. 


Hover 

Mean  Deviation 

o-value  (p  <  .C 

15) 

Measure 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Position  (feet) 

Legacy 

(Lateral  Hover  Position 
RMSD) 

6.353 

6.646 

5.958 

5.819 

0.786 

0.396 

0.414 

0.722 

0.519 

0.968 

BOSS 

(Lateral  Hover  Position 
RMSD) 

1.335 

1.533 

1.432 

2.481 

0.452 

0.710 

0.531 

0.687 

0.810 

0.677 

FISH 

(Lateral  Hover  Position 
RMSD) 

1.644 

1.878 

1.543 

2.069 

0.260 

0.950 

0.478 

0.200 

0.925 

0.494 

Legacy 

(Longitudinal  Hover 
Position  RMSD) 

20.989 

25.780 

23.858 

18.383 

0.327 

0.473 

0.685 

0.519 

0.103 

0.061 

BOSS 

(Longitudinal  Hover 
Position  RMSD) 

2.620 

3.547 

3.324 

3.311 

0.941 

0.993 

1.000 

0.943 

0.940 

0.994 

FISH 

(Longitudinal  Hover 
Position  RMSD) 

12.056 

10.917 

11.363 

15.959 

0.778 

0.872 

0.586 

0.940 

0.699 

0.643 

Heading  (degrees) 

Legacy 

(Hover  RMSD) 

2.381 

2.063 

1.917 

1.749 

0.907 

0.573 

0.729 

0.809 

0.793 

0.896 

BOSS 

(Hover  RMSD) 

1.467 

2.289 

1.499 

2.308 

0.662 

0.633 

0.928 

0.223 

0.679 

0.839 

fish’ 

(Hover  RMSD) 

0.944 

0.834 

0.753 

1.535* 

0.587 

0.215 

0.661 

0.251 

0.385 

0.026 

Altitude  (feet) 

Legacy 

(Hover  RMSD) 

3.787 

3.644 

3.288 

3.586 

0.945 

0.912 

0.840 

0.794 

0.949 

0.720 

BOSS 

(Hover  RMSD) 

1.173 

1.027 

1.049 

0.998 

0.481 

0.774 

0.695 

0.525 

0.970 

0.657 

FISH 

(Hover  RMSD) 

2.652 

2.907 

2.819 

3.698 

0.790 

0.898 

0.199 

0.939 

0.422 

0.134 

*  “Both”  refers  to  the  combined  TSAS  and  Aural  display. 
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Table  A-ll. 

Summary  of  subjective  data  for  symbology  for  Hover  maneuver. 


Hover 

Mean 

/j-value  (p  <  .05) 

Measure 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Cooper  Harper 

5.1875 

3.5313 

4.562 

0.018 

0.398 

0.042 

Bedford 

5.3125 

3.7188 

5.0313 

0.020 

0.779 

0.027 

VCI  Attitude 

3.6878 

2.2497 

2.7288 

0.012 

0.161 

0.205 

VCI  Horizontal 

3.1875 

1.9375 

2.5938 

0.012 

0.293 

0.125 

VCI  Vertical 

3.500 

1.5938 

2.5625 

0.012 

0.074 

0.017 

Table  A- 12. 

Summary  of  subjective  data  for  cueing  for  Hover  maneuver. 


Hover 

Mean 

p-value  (p  < .( 

>5) 

Measure 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Cooper  Harper 

4.375 

4.417 

4.458 

4.458 

0.915 

0.724 

0.500 

0.670 

0.734 

1.000 

Bedford 

4.625 

4.750 

4.750 

4.625 

0.596 

0.490 

0.799 

0.932 

0.609 

0.395 

VCI  Attitude 

2.958 

2.972 

2.820 

2.806 

0.888 

0.483 

0.528 

0.248 

0.397 

0.893 

VCI  Horizontal 

2.583 

2.500 

2.708 

2.500 

0.492 

0.496 

0.751 

0.059 

0.715 

0.344 

VCI  Vertical 

2.583 

2.667 

2.500 

2.458 

0.495 

0.526 

0.778 

0.553 

0.339 

0.786 

*  “Both”  refers  to  t 

le  combinec 

TSAS  and  Aural  display. 

Table  A- 13. 

Mean  deviations  by  symbology  and  significance  values  between  symbology  sets  for  Sidestep 

maneuver. 


Sidestep 

Mean 

o-value  (p  <  .C 

•5) 

Index 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Position(feet) 

80.340 

12.160 

13.015 

<0.001 

<0.001 

0.763 

Heading  (degrees) 

1.991 

1.895 

0.989 

1.000 

0.078 

0.069 

Altitude  (feet) 

7.035 

5.035 

5.280 

0.642 

0.854 

0.981 

Speed  (knots) 

5.066 

5.640 

3.987 

<0.001 

<0.001 

<0.001 
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Table  A- 14. 

Cueing  mean  deviations  and  significance  values  for  Sidestep  maneuver. 


Sidestep 

Mean  Deviation 

o-value  (p  < .( 

15) 

Measure 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Position  (feet) 

Legacy 

(Lateral  Position  RMSD) 

78.442 

75.674 

81.210 

86.034 

0.847 

0.552 

0.334 

0.452 

0.231 

0.261 

BOSS 

(Lateral  Position  RMSD) 

11.883 

12.686 

10.648 

13.423 

0.855 

0.619 

0.757 

0.776 

0.712 

0.519 

FISH 

(Lateral  Position  RMSD) 

10.064 

6.695 

13.122 

22.180 

0.134 

0.225 

0.019 

0.017 

0.011 

0.305 

Heading  (degrees) 

Legacy 

(Sidestep  RMSD) 

2.047 

1.892 

1.796 

2.230 

0.639 

0.850 

0.723 

0.636 

0.838 

0.781 

BOSS 

(Sidestep  RMSD) 

1.335 

2.240 

1.739 

2.267 

0.407 

0.386 

0.838 

0.889 

0.787 

0.835 

FISH 

(Sidestep  RMSD) 

0.838 

1.157 

0.806 

1.154 

0.258 

0.131 

0.488 

0.017 

0.635 

0.074 

Altitude  (feet) 

Legacy 

(Maximum  Altitude) 

33.685 

34.234 

33.443 

33.659 

0.916 

0.805 

0.869 

0.834 

0.869 

0.941 

BOSS 

(Maximum  Altitude) 

33.778 

33.132 

33.865 

32.591 

0.483 

0.833 

0.201 

0.398 

0.304 

0.108 

FISH 

(Maximum  Altitude) 

31.345 

32.235 

31.792 

33.217 

0.325 

0.545 

0.366 

0.721 

0.682 

0.481 

Legacy 

(Minimum  Altitude) 

26.379 

27.423 

27.902 

25.179 

0.187 

0.202 

0.393 

0.783 

0.171 

0.709 

BOSS 

(Minimum  Altitude) 

28.572 

27.036 

28.456 

29.161 

0.294 

0.960 

0.488 

0.325 

0.191 

0.341 

FISH 

(Minimum  Altitude) 

26.687 

25.896 

26.983 

27.904 

0.553 

0.828 

0.243 

0.471 

0.258 

0.591 

Speed  (knots) 

Legacy 

(Maximum  Velocity) 

5.040 

5.352 

4.988 

4.885 

0.133 

0.832 

0.150 

0.010 

0.013 

0.506 

BOSS 

(Maximum  Velocity) 

5.645 

5.651 

5.853 

5.411 

0.973 

0.421 

0.401 

0.246 

0.537 

0.064 

FISH 

(Maximum  Velocity) 

3.939 

3.948 

3.853 

4.206 

0.963 

0.312 

0.285 

0.181 

0.404 

0.237 

‘Both”  refers  to  the  combined  TSAS  and  Aural  disp 


ay. 
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Table  A- 15. 

Summary  of  subjective  data  for  symbology  for  Sidestep  maneuver. 


Sidestep 

Mean 

o-value  (p  <  .05) 

Measure 

Legacy 

BOSS 

FISH 

Legacy/ 

BOSS 

Legacy/ 

FISH 

FISH/ 

BOSS 

Cooper  Harper 

4.969 

4.156 

4.313 

0.079 

0.395 

0.799 

Bedford 

5.375 

4.500 

4.969 

0.079 

0.528 

0.181 

VCI  Attitude 

3.740 

2.614 

2.646 

0.012 

0.123 

0.499 

VCI  Horizontal 

3.063 

2.063 

2.344 

0.011 

0.233 

0.671 

VCI  Vertical 

3.469 

2.031 

2.531 

0.017 

0.150 

0.105 

Table  A- 16. 

Summary  of  subjective  data  for  cueing  for  Sidestep  maneuver. 


Sidestep 

Mean 

p-value  (p  < .( 

>5) 

Measure 

IR 

TSAS 

Aural 

Both* 

IR/ 

TSAS 

IR/ 

Aural 

IR/ 

Both* 

TSAS/ 

Aural 

TSAS/ 

Both* 

Aural/ 

Both* 

Cooper  Harper 

4.417 

4.500 

4.583 

4.417 

0.750 

0.157 

0.596 

0.730 

0.609 

0.465 

Bedford 

4.958 

4.958 

4.875 

5.000 

0.684 

0.497 

0.865 

0.458 

0.932 

0.339 

VCI  Attitude 

3.069 

3.028 

2.903 

3.000 

0.917 

0.352 

0.575 

0.465 

0.854 

0.465 

VCI  Horizontal 

2.667 

2.500 

2.333 

2.458 

0.244 

0.027 

0.064 

0.180 

0.496 

0.461 

VCI  Vertical 

2.792 

2.667 

2.583 

2.667 

0.480 

0.344 

0.553 

1.000 

1.000 

0.726 

*  “Both”  refers  to  t 

le  combinec 

TSAS  and  Aural  display. 
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Appendix  B. 
Questionnaire 


Subject  #:  Date: 

Flight  hours  experience 

Total  flight  hours:  _  Pilot  in  Command:  _ 

UH-60 _  Other  Rotary _ 

NVS  (FLIRs) _  Other  aircraft _ 

NVG: _ 

Experience  with  cueing  technologies:  (hours) 

Legacy  HUD  : _  FISH:  _ 

BOSS:  _  TSAS: _  Aural  Cueing  Products: _ 

Task  Difficulty  Ranking 

1 .  Please  rank  order  the  difficulty  of  the  three  different  maneuvers  performed  while  using  the 
multifunctional  display  with  visual  symbology,  haptic  cues,  and  aural  cues,  where  1  is  the  least 
difficult  and  4  is  the  most  difficult.  Thus,  each  column  will  rank  order  the  difficulty  of  the  three 
flight  tasks  for  that  column’s  symbology/cue  set.  (circle  one  in  each  category) 


Category 

Approach 

and 

Landing 

Hover 

Sidestep 

Legacy  HUD 

1234 

1234 

1234 

Legacy  + 

Haptic 

1234 

1234 

1234 

Legacy  +  Aural 

1234 

1234 

1234 

Legacy  +  Aural  + 
Haptic 

1234 

1234 

1234 

Category 

Approach 

and 

Landing 

Hover 

Sidestep 

BOSS 

12  34 

1234 

1234 

BOSS  +  Haptic 

1234 

1234 

1234 

BOSS  +  Aural 

1234 

1234 

1234 

BOSS  +  Aural  + 

Haptic 

1234 

1234 

1234 
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Category 

Approach 

and 

Landing 

Hover 

Sidestep 

FISH 

1234 

1234 

12  34 

Fish  +  Haptic 

1234 

1234 

1234 

FISH  +  Aural 

1234 

1234 

1234 

FISH  +  Aural  + 
Haptic 

1234 

1234 

1234 

2.  What  visual  symbology  indications  do  you  feel  are  necessary  on  the  multifunction  display  in  order 
to  successfully  conduct  the  test  maneuvers  (check  all  that  apply)? 

_ Airspeed  _ Vertical  Speed 

_ Attitude  _ Velocity  Vector  _ Torque 

_ Heading  _ Turn  &  Slip  _ Altimeter 

3.  What  advanced  cueing  indications  do  you  feel  are  beneficial  in  order  to  adequately  conduct  all  test 
maneuvers  (check  all  that  apply)? 

Haptic: 

_ Airspeed  _ Vertical  Speed 

_ Attitude  _ Velocity  Vector  _ Torque 

_ Heading  _ Turn  &  Slip  _ Altitude 

Aural: 

_ Airspeed  _ Vertical  Speed 

_ Attitude  _ Velocity  Vector  _ Torque 

_ Heading  _ Turn  &  Slip  _ Altitude 

4.  What  other  combinations  of  cueing  technologies  would  you  desire  to  see  implemented  in  the  future? 
(Please  describe  below) 


Thank  you  for  your  time  and  service. 
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Appendix  C. 

Outlier  charts. 


Approach  to  landing 


Approach  to  Landing-  Distance  at  Touch  Down 
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Figure  C-l.  Approach  to  Landing-  Distance  at  Touch  Down. 
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Figure  C-2.  Approach  to  Landing-  Level  Segment  Heading  Standard  Deviation. 
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Approach  to  Landing-  Transition  to  Approach  Heading  Standard 
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Figure  C-3.  Approach  to  Landing-  Transition  to  Approach  Fieading  Standard  Deviation. 
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Figure  C-4.  Approach  to  Landing-  Approach  Heading  Standard  Deviation. 
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Figure  C-5.  Approach  to  Landing-  Transition  to  Touchdown  Heading  Standard  Deviation. 
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Approach  to  Landing-  Level  Segment  Altitude  Root-Mean-Square 

Deviation 
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Figure  C-6.  Approach  to  Landing-  Level  Segment  Altitude  Root-Mean-Square  Deviation. 
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Figure  C-7.  Approach  to  Landing-  Level  Segment  Speed  Root-Mean-Square  Deviation. 
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Approach  to  Landing-  Approach  Segment  Speed  Root-Mean-Square 

Deviation 
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Figure  C-8.  Approach  to  Landing-  Approach  Segment  Speed  Root-Mean-Square  Deviation. 
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Approach  to  Landing-  U  (Longitudinal)  Speed  at  Touch  Down 
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Figure  C-9.  Approach  to  Landing-  U  (Longitudinal)  Speed  at  Touch  Down. 
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Figure  C-10.  Approach  to  Landing-  V  (Lateral)  Speed  at  Touch  Down. 
Approach  to  hover 
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Approach  to  Hover-  Hover  Lateral  Position  Root-Mean-Square 

Deviation 
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Figure  C-l  1.  Approach  to  Hover-  Hover  Lateral  Position  Root-Mean-Square  Deviation. 
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Approach  to  Hover-  Hover  Longitudinal  Position  Root-Mean-Square 

Deviation 
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Figure  C-12.  Approach  to  Hover-  Hover  Longitudinal  Position  Root-Mean-Square  Deviation. 
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Figure  C-13.  Approach  to  Hover-  Hover  Heading  Root-Mean-Square  Deviation. 
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Approach  to  Hover-  Level  Segment  Heading  Standard  Deviation 
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Figure  C-14.  Approach  to  Hover-  Level  Segment  Heading  Standard  Deviation. 
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Figure  C-15.  Approach  to  Hover-  Transition  to  Approach  Heading  Standard  Deviation. 
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Figure  C-16.  Approach  to  Hover-  Approach  Heading  Standard  Deviation. 
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Approach  to  Hover-  Transition  to  Hover  Heading  Standard 
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Figure  C-17.  Approach  to  Hover-  Transition  to  Hover  Heading  Standard  Deviation. 
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Approach  to  Hover-  Level  Segment  Altitude  Root-Mean-Square 

Deviation 
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Figure  C-18.  Approach  to  Hover-  Level  Segment  Altitude  Root-Mean-Square  Deviation. 
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Approach  to  Hover-  Hover  Altitude  Root-Mean-Square  Deviation 
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Figure  C-19.  Approach  to  Hover-  Hover  Altitude  Root-Mean-Square  Deviation. 
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Figure  C-20.  Approach  to  Hover-  Level  Segment  Speed  Root-Mean-Square  Deviation. 
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Approach  to  Hover-  Approach  Segment  Speed  Root-Mean-Square 

Deviation 
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Figure  C-21.  Approach  to  Hover-  Approach  Segment  Speed  Root-Mean-Square  Deviation. 
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Hover-  Heading  Root-Mean-Square  Deviation 
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Figure  C-22.  Hover-  Heading  Root-Mean-Square  Deviation. 
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Hover-  Lateral  Position  Root-Mean-Square  Deviation 
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Figure  C-23.  Hover-  Lateral  Position  Root-Mean-Square  Deviation. 
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Hover-  Longitudinal  Position  Root-Mean-Square  Deviation 
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Figure  C-24.  Hover-  Longitudinal  Position  Root-Mean-Square  Deviation. 
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Figure  C-25.  Hover-  Altitude  Root-Mean-Square  Deviation. 
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Sidestep 
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Figure  C-26.  Sidestep-  Lateral  Position  Root-Mean-Square  Deviation. 
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Figure  C-27.  Sidestep-  Heading  Root-Mean-Square  Deviation. 
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Figure  C-28.  Sidestep-  Maximum  Altitude. 
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Sidestep-  Minimum  Altitude 


35 

30 

d  25 

27  20 

■o 

3  15 

*  10 


Jlimuki.H'llifcnIillHiMJih  il  m 

I  li  II  i  i 

l  I 

I  I  1  I  I  I  I  I  I  I  I  I 
III  1111  I  I II  II  I  I I  I 


1  2  3  4  5  6  7 

Cueing  Condition 


9  10  11  12 


■  Pilot  1 

■  Pilot  2 

■  Pilot  3 

■  Pilot  4 
V)  Pilot  5 

■  Pilot  6 

■  Pilot  7 

■  Pilot  8 


Figure  C-29.  Sidestep-  Minimum  Altitude. 
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Figure  C-30.  Sidestep-  Maximum  Lateral  Speed. 
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Appendix  D. 
Biometric  results. 


Biometric  Output  Means  (by  Measure,  Condition,  and  Maneuver) 


1 )  Mean  Heart  Rate  (BPM)  (by  Condition  and  Maneuver) 


AppHOV 

AppLND 

HOV 

Side 

1 

81.58143  (6.34) 

83.48814  (5.99) 

78.51386  (7.58) 

82.03614  (8.48) 

2 

81.38643  (6.92) 

83.75167  (6.12) 

81.10533  (8.75) 

81.81417(7.06) 

3 

80.77800  (4.94) 

82.11829  (6.58) 

77.62957  (5.45) 

79.69300  (4.89) 

4 

82.47700  (7.72) 

84.62933  (4.26) 

81.40467  (6.24) 

80.66729  (7.91) 

5 

79.19300  (4.93) 

80.03400  (5.35) 

78.96514  (5.95) 

79.50157  (4.73) 

6 

81.61200  (2.24) 

83.03833  (4.49) 

81.77329  (3.55) 

82.27029  (4.44) 

7 

81.69800  (7.07) 

84.08043  (5.99) 

82.66671  (6.57) 

84.45543  (8.68) 

8 

80.54857  (8.22) 

83.60557  (6.53) 

81.10929  (7.99) 

85.07614  (5.66) 

9 

76.59514  (6.02) 

81.97200  (8.49) 

75.49071  (5.25) 

77.03871  (5.84) 

10 

80.89680  (6.38) 

82.90867  (6.15) 

79.12060  (8.44) 

82.62850  (8.03) 

11 

82.13900  (6.24) 

81.14986  (6.94) 

80.97286  (6.26) 

82.89929  (5.71) 

12 

77.63783  (6.38) 

81.11060  (6.76) 

79.69514  (6.99) 

79.26500  (9.53) 

2)  Mean  Respiration 

Intervals  (s)  (by  Condition  and  Maneuver) 

AppHOV 

AppLND 

HOV 

Side 

1 

1.510000  (0.17) 

1.534875  (0.16) 

1.469250  (0.18) 

1.463000  (0.09) 

2 

1.546500  (0.06) 

1.590625  (0.18) 

1.620750  (0.13) 

1.660000  (0.09) 

3 

1.522625  (0.11) 

1.564375  (0.13) 

1.567125  (0.11) 

1.486875  (0.09) 

4 

1.521375  (0.15) 

1.548250  (0.11) 

1.569000  (0.12) 

1.544625  (0.09) 

5 

1.573750  (0.11) 

1.568625  (0.09) 

1.585125  (0.12) 

1.574500  (0.08) 

6 

1.532000  (0.13) 

1.551429  (0.06) 

1.575625  (0.13) 

1.519125  (0.14) 

7 

1.526625  (0.11) 

1.551375  (0.11) 

1.538375  (0.13) 

1.541000  (0.12) 

8 

1.522625  (0.11) 

1.524625  (0.12) 

1.552125  (0.13) 

1.528250  (0.14) 

9 

1.561625  (0.11) 

1.527375  (0.11) 

1.611875  (0.11) 

1.581875  (0.17) 

10 

1.511500  (0.12) 

1.488250  (0.11) 

1.563000  (0.11) 

1.586875  (0.14) 

11 

1.550875  (0.12) 

1.536875  (0.13) 

1.565375  (0.12) 

1.612375  (0.12) 

12 

1.561875  (0.14) 

1.502750  (0.15) 

1.564500  (0.06) 

1.606571  (0.09) 
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3)  Mean  Tonic  GSR  (mS)  (by  Condition  and  Maneuver) 


AppHOV 

AppLND 

HOV 

Side 

1 

5.989375  (5.77) 

5.190714  (6.64) 

5.484250  (5.32) 

5.464375  (5.33) 

2 

4.326667  (2.09) 

4.931667  (1.66) 

4.505000  (1.81) 

4.853667  (1.74) 

3 

8.912000  (7.82) 

7.142857  (7.01) 

8.574400  (7.42) 

9.282800  (7.95) 

4 

6.903714  (4.81) 

3.540000  (2.87) 

6.882429  (5.50) 

6.888429  (5.67) 

5 

6.132250  (4.95) 

6.501000  (5.38) 

5.443875  (4.84) 

3.716857  (3.33) 

6 

4.000000  (2.18) 

3.338333  (2.15) 

3.751667  (2.14) 

2.631429  (2.16) 

7 

5.492833  (4.26) 

4.013333  (2.53) 

3.042000  (2.41) 

3.732600  (2.55) 

8 

5.509250  (5.62) 

5.896500  (5.75) 

3.923750  (2.14) 

4.317125  (2.41) 

9 

3.775571  (5.96) 

2.384286  (2.34) 

3.467143  (2.49) 

3.235714  (2.67) 

10 

3.452857  (2.25) 

4.668571  (1.95) 

3.707143  (2.50) 

3.861667  (1.83) 

11 

4.064714  (2.80) 

3.084286  (2.43) 

3.634000  (2.77) 

3.943875  (3.29) 

12 

4.183000  (1.84) 

3.813333  (2.28) 

4.371500  (2.02) 

4.072833  (2.41) 

4)  Mean  Sympathetic  HRY  Ratio  (by  Condition  and  Maneuver) 


AppHOV 

AppLND 

HOV 

Side 

1 

4.068571  (5.45) 

2.558571  (2.33) 

4.890000  (5.06) 

1.2742857  (1.07) 

2 

3.975714  (4.58) 

4.725000  (7.74) 

3.611667  (4.89) 

0.8066667  (0.68) 

3 

1.864286  (1.36) 

1.958571  (2.36) 

1.784286  (1.10) 

1.1957143  (1.40) 

4 

5.248333  (3.51) 

4.038333  (3.64) 

6.408333  (11.25) 

1.0814286  (1.34) 

5 

3.677143  (2.74) 

5.290000  (6.54) 

7.157143  (11.34) 

1.0100000  (0.71) 

6 

4.734000  (6.71) 

4.261667  (8.04) 

2.722857  (3.13) 

0.8357143  (0.65) 

7 

3.950000  (3.9) 

2.418571  (2.18) 

2.857143  (2.31) 

1.3800000  (1.62) 

8 

4.628571  (8.78) 

1.788571  (1.89) 

4.577143  (5.42) 

1.1400000  (1.55) 

9 

2.855714  (1.95) 

2.340000  (1.74) 

2.371429  (1.63) 

0.8971429  (0.62) 

10 

2.898000  (4.09) 

4.110000  (3.32) 

6.572000  (7.11) 

1.9375000  (1.28) 

11 

2.235714  (2.19) 

3.721429  (5.51) 

4.614286  (6.53) 

1.2271429  (1.15) 

12 

1.848333  (1.69) 

2.174000  (2.52) 

1.734286(1.40 

1.1366667  (1.63) 

5)  Summary  of  Conditions  with  Lowest  Biometric  Stress  Responses  by  Maneuver  and 
Measure 


Mean  HR 

Mean  Resp.  Int. 

Mean  GSR 

HRV  Ratio 

AppHOV 

FISH+IR  (9) 

BOSS+IR  (5) 

FISH+IR+TSAS  (10) 

FISH+IR+TSAS+Auditory 

(12) 

AppLND 

BOSS+IR  (5) 

Legacy+TSAS  (2) 

FISH+IR  (9) 

BOSS+IR+TSAS+Auditory 

(8) 

HOV 

FISH+IR  (9) 

Legacy+TSAS  (2) 

BOSS+Auditory  (7) 

FISH+IR+TSAS+Auditory 

(12) 

Side 

FISH+IR  (9) 

Legacy+TSAS  (2) 

BOSS+TSAS  (6) 

Legacy+TSAS  (2) 
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Multivariate  Analysis  of  Variance:  All  Biometrics  by  Condition  and  Controlling  for 
Maneuver  Effects 

(p  <  0.05) 


DF 

Wilks 

Approx  F 

Num  DF 

Den  DF 

Significance 

Maneuver 

3 

0.14125 

2.61864 

12 

636 

0.002 

Condition 

11 

0.40716 

2.19437 

44 

852 

<.001 

Man  *  Cond 

33 

0.26453 

0.45709 

132 

852 

1.000 

Residuals 

213 

Summary:  Even  when  correcting  for  the  significant  effect  of  Maneuver  on  Biometric 
Outcomes,  there  is  also  a  significant  effect  for  Condition.  The  interaction  between  Maneuver  and 
Condition  is  nonsignificant,  which  suggests  that  the  effects  of  the  Condition  were  consistent 
across  Maneuvers. 

Univariate  Breakdowns  of  Condition  Effects,  Controlling  for  Maneuver  Effects 

123  observations  deleted  due  to  missing  data  (p  <  0.05) 


Measure 

DF 

Sum  Sq 

Mean  Sq 

F  Value 

Significance 

Mean  HR 

Maneuver 

3 

225.8 

75.265 

1.6364 

0.182 

Condition 

11 

922.7 

83.878 

1.8237 

0.052 

Man  :  Cond 

33 

429.5 

13.016 

0.2830 

0.999 

Residuals 

213 

9796.7 

45.994 

Mean  Resp 

Maneuver 

3 

0.05408 

0.018026 

1.1227 

0.301 

Condition 

11 

0.47616 

0.043287 

2.9465 

0.001 

Man  :  Cond 

33 

0.33803 

0.010243 

0.6973 

0.999 

Residuals 

213 

3.12913 

0.014691 

Mean  GSR**  (Lots  of  missing  data  due  to  movement 
artifacts  from  hand-mounted  electrodes) 

Maneuver 

3 

26.3 

8.778 

0.4547 

0.714 

Condition 

11 

719.2 

65.379 

3.3865 

<.001 

Man  :  Cond 

33 

200.0 

6.061 

0.3139 

0.999 

Residuals 

213 

4112.2 

19.306 

HRV  Ratio 

Maneuver 

3 

351.9 

117.302 

5.2179 

0.002 

Condition 

11 

172.9 

15.720 

0.6993 

0.739 

Man  :  Cond 

33 

297.2 

9.005 

0.4006 

0.999 

Residuals 

213 

4788.4 

22.481 
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Condition-Specific  Effects  for  Each  Significant  Biometric  Outcome  Variable  (Using 
Tukey  Honestly  Significant  Difference  Post  Hoc  Tests) 


1)  Mean  Respiration  Interval 

Pairwise  post  hoc  tests  of  the  main  effect  for  Condition  on  Mean  Respiration  Interval 
(controlling  for  the  non-significant  effect  of  Maneuver),  demonstrated  the  following  significant 
pairwise  differences: 

Condition  2  >  Condition  1  (Diff  =  0.1 1  ,/?  =  0.018) 

Condition  1  demonstrated  a  significantly  lower  mean  inter-peak-interval  for  respiration  than 
Condition  2,  which  suggests  that  pilots’  stress  levels  were  significantly  lower  in  Condition  2  than 
in  Condition  1 . 

2)  Mean  GSR 

Pairwise  post  hoc  tests  of  the  main  effect  for  Condition  on  Mean  Tonic  GSR  (controlling  for 
the  nonsignificant  effect  of  Maneuver),  demonstrated  the  following  significant  pairwise 
differences: 

Condition  3  >  Condition  6  (Diff  =  4.966,  p  =  0.002) 

Condition  3  >  Condition  7  (Diff  =  4.251,/?  =  0.028) 

Condition  3  >  Condition  9  (Diff  =  5.156,/?  <  .001) 

Condition  3  >  Condition  10  (Diff  =  4.454,/?  =  0.008) 

Condition  3  >  Condition  1 1  (Diff  =  4.675,/?  =  0.004) 

Condition  3  >  Condition  12  (Diff  =  4.262,/?  =  0.021) 

Condition  3  demonstrated  a  Tonic  GSR  level  that  was  significantly  higher  than  several  other 
Conditions,  indicating  that  pilots’  stress  levels  were  significantly  higher  in  this  condition  than 
Conditions  6,  7,  9,  10,  11,  and  12.  The  differences  between  Condition  3  and  Condition  2  (3.718) 
and  Condition  3  and  Condition  8  (3.460)  were  approaching  significance  (/?  <  0.1). 


3)  **Mean  HR  was  approaching  significance  (p  =  0.05) 

Data  from  EKG  were  extremely  consistent  and  strong,  and  while  not  as  sensitive  to  condition 
shifts  in  an  extremely  small  population  (N=  8).  Increased  sample  sizes  may  see  significant 
results,  and  data  would  be  among  the  most  reliable  in  the  biometric  series.  The  difference 
between  conditions  that  was  approaching  significance  was  between  Condition  7  and  Condition  9 
(5.451,/?  =  0.083),  which  would  indicate  that  pilots’  stress  levels  were  higher  in  Condition  7  than 
they  were  in  Condition  9. 
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Maneuver-Specific  Effects  for  Each  Significant  Biometric  Outcome  Variable 

1)  HRY  Sympathetic:  Vagal  Proportion  Ratio 

While  not  sensitive  to  Condition  effects  in  the  present  test,  HRV  appeared  to  be  quite 
sensitive  to  changes  between  Maneuvers.  Like  HR,  HRV  data  is  extracted  from  the  ECG  signal, 
which  was  consistently  the  strongest  and  most  accurate  data  collected  (it  was  very  resistant  to 
movement  artifact).  HRV  is  divided  into  two  proportion  categories  based  on  a  spectral  density 
analysis  of  frequencies;  these  categories  are  Sympathetic  Activity  (correlates  with  higher  stress 
levels)  and  Vagal  Activity  (correlates  with  lower  stress  levels).  A  ratio  of  the  proportion  of 
variance  attributed  to  Sympathetic  Activity  to  the  variance  attributed  to  Vagal  Activity  provides 
a  single  measure  of  HRV:  higher  levels  of  this  ratio  indicate  higher  levels  of  stress. 

A  post  hoc  analysis  of  the  significant  effect  for  Maneuver  on  HRV  Ratio  indicated  the 
following  significant  differences: 

Side  <  HOV  (Diff  =  2.889 ,p<  .001) 

Side  <  AppHOV  (Diff  =  2.346, =  0.004) 

Side  <  AppLND  (Diff  =  3.908,;?  =  0.016) 

The  Side-Step  maneuver  demonstrated  a  significantly  lower  HRV  Ratio  than  all  other 
maneuvers,  which  would  suggest  that  pilots’  stress  levels  were  lowest  while  completing  this 
maneuver. 
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